I think you don't use Mutex correctly, take a look at the documentation:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682411%28v=vs.85%29.aspx[
^]
Return value
If the function succeeds, the return value is a handle to the newly created mutex object.
If the function fails, the return value is NULL. To get extended error information, call GetLastError
.
That means you need something like this:
BOOL COneInstanceDlg::OnInitDialog()
{
...
HANDLE hMutex;
hMutex=CreateMutex(NULL,true,_T("Test program"));
if(NULL == hMutex)
{
if(ERROR_ALREADY_EXISTS==GetLastError())
{
ExitProcess(0); }
}
return TRUE;
}
P.S.: Apart of this I would make m_hMutex a member variable of your COneInstanceDlg class to have the chance to release the mutex after your program exits.