You cannot do it in the application setting up the registry. You can only do it in the target application itself, the one you want to be executed in a single instance.
As you did not tag the language and the platform (always do it), I'll just explain the idea.
By definition, OS can start the application from its executable module any number of type. Each time it will create a separate process. The processes are isolated, run in isolated address spaces and "know" nothing about each other. So, you need to use some IPC to make them collaborate:
http://en.wikipedia.org/wiki/Inter-process_communication[
^].
There are different IPC mechanisms used to solve your problem, and most of such solutions will work. In my last solutions, I uses named pipes, through .NET remoting. I'll explain why below.
Your goal is to allow at most two process instances of the same application at a time. Two, not one, one makes no sense; if there is one, another one can always be started, nothing prevents the OS to start it. But now, when a second one is started, it should try to detect the first instance. If previously existing instance is detected, the second one should terminate itself immediately. Can you see that it solves the problem?
Now, I promised to explain why did I use named pipes. This is because it is often required that the second instance gets its command line parameter and passes this data to the first one. As you can see in many single-instance application, if you specify, say, a file, it is still opened in a first instance when you try to start a second one. The pipe transmits the collection of command line parameters to the first instance. Here is the sequence: tries to connect to existing first instance of the application. If this attempt has failed, this instance is the first one. The application should create the named pipe and then it should proceed in a "regular" way. If the attempt to connect succeeds (the pipe was already created by a first instance), it means the current application is the second one. Good, the application should get command line parameters, pass it the pipe and terminate itself immediately after completion of passing data.
Now, the final hint. All IPC will need some unique string
Atom for unique identification used between the processes. How to get one? If you use something like application name, nothing guarantees that some unrelated application does not use the same. What string to use? I use the location (path name) of the entry-point executable module of the application. However, it allows to execute excessive process if you copy the whole application to a different location, but I consider it as acceptable, or even beneficial behavior.
[EDIT]
See also my past answer:
Enter multiple commands to only one process instance[
^].
Too bad you did not specify your platform/language in first place.
Using .NET justifies using "classical" .NET remoting with "IPC" channel. In remoting terminology, "IPC" means the channel based in named pipes, but you don't create a named pipe directly.
Your application should try to play the role of either client-side or server-side remote object trying to create one. This mechanism serves dual purpose: it detects if the instance is first or not, and it also provides a channel for passing data. This dual-purpose approach allows for extremely compact solution.
First, your application tries to connect to a remote object. It may fail, the means the current instance is the first one. Then you recover from exception and try to create a remote object. If connection is successful, this is the second instance. Pass data using the connection and terminate immediately.
—SA
Submit an article or tip