When an executable is open in the system... there will be a process with the exact same name as the file. For example if you are looking for an exe file which is called: SystemCleaner.exe you can find it as a process with the name: SystemCleaner.
Accordingly if the file name has a space, for example: System Cleaner.exe Then you will looking for a process called: System Cleaner.
You can count how many processes with a specific name exist in the system.. with the following lines of code(in C#):
int counter = 0;
foreach (Process process in Process.GetProcessesByName("SystemCleaner"))
{
counter++;
}
if(counter > 0)
{
YourButtonName.Visible = true;
}
else
{
YourButtonName.Visible = false;
}