Instead of a Mutex you could use a ManualResetEvent and have the first instance register a thread pool wait on it. When another instance starts it can attempt to open up the MRE, if it exists then it can set it which will signal the first instance to close and then reset it and then register it's thread pool wait on it.
Something like this:
bool createdNew = false;
EventWaitHandle instanceEvent = new EventWaitHandle(false, EventResetMode.ManualReset, "MyInstanceEvent", out createdNew);
if (!createdNew)
{
instanceEvent.Set();
}
instanceEvent.Reset();
ThreadPool.RegisterWaitForSingleObject(instanceEvent, (state, timedOut) =>
{
instanceEvent.Close();
this.Close();
}, null, Timeout.Infinite, true);