How can I write a Self-Modifying code ic C?

We know what self-modifying code is, and it's not possible in C.

0) Each compiler generates a slightly different binary representation of the source code (even between versions of the same compiler).

1) You would have to disassemble the code - in memory - and then modify the assembly code, and then somehow inject it back into the running application, and then jump to the new code at the appropriate position.

2) Function pointers is the best you can hope for, and even those are static once you compile the code. I suppose you could create a function pointer that jumps outside of the current code and executes other code outside of the currently running application, but that's too close to deliberate comprimise of the security of the system in question, which is what I suspect you're trying to accomplish.

It depends what you call "C". If you mean any of its concrete implementation, a relevant part of it is inline assembler. If you mean standard C or any other common sub-set of the language dialects and mean portable code, inline assembler and the rest of my post is of course irrelevant.

Writing a self-modification code in C would be quite possible using its feature called inline assembler (http://en.wikipedia.org/wiki/Inline_assembly[^]).

As with assembly language anyone can try to right anything which can be done with the CPU, the question is reduced to the question: is writing a self-modifying code possible in principle.

The answer is: it depends on the type of the CPU and operating system kernel.

To see some examples of this activity: http://stackoverflow.com/questions/4812869/how-to-write-self-modifying-code-in-x86-assembly[^], http://asm.sourceforge.net/articles/smc.html[^], http://www.programmingforums.org/post205520.html[^].

I used to write self-modifying code in MS-DOS using inline assembler (as well as the stand-along Assembler) without much problems. This system uses real-mode of x86 CPUs (or CPUs not having protected mode in principle). It was also possible with Windows 95/98.

With the real-mode of Intel CPU, it is possible to guard some segment of code from writing forever, oven for the code in inner CPU protection ring. It's even easier to protect all executable segments from writing by the code running in outer rings. In this way, only the system kernel can create and/or modify executable segments of memory.

Franky, the real-life situation with existing modern operating systems remains open to me.

I've tryied to do this in modern NT-based Windows or Linux. I just tried a naive write into a memory occupied by really existing function. As I much expected, it causes exception "Access violation writing location…". However, this is not the only way to access memory. More advanced way is an attempt to create an alias memory segment pointing to the same linear memory location and give it read/write access. At the level of OS API, one of the possible exploits can be writing a file with I/O access mapping to the protected memory location. The OS could be designed totally protected from all such exploits. I don't have reliable information on the real-life situation. Even if vulnerability of this kind could be known to the team developing a commercial OS, I would expect it is kept as top secret.

About self-modifying code, read http://en.wikipedia.org/wiki/Self-modifying_code#Operating_systems[^].

On protected mode, see http://en.wikipedia.org/wiki/Protected_mode[^].

I agree with Griff and John on the practical applicability of the concept of self-modifying code. I think OS should provide 100% protection (for the CPU allowing such protection) from any attempts to modify executable code in memory and leave no back doors for any code.

—SA

Here is an explaination of why it's not a good idea these days Data Execution Prevention[^]

RegisterServiceCtrlHandler has the following signature

SERVICE_STATUS_HANDLE WINAPI RegisterServiceCtrlHandler(
  __in  LPCTSTR lpServiceName,
  __in  LPHANDLER_FUNCTION lpHandlerProc
);

and it has bee superseeded by RegisterServiceCtrlHandlerEx, introduced with Windows 2000

SERVICE_STATUS_HANDLE WINAPI RegisterServiceCtrlHandlerEx(
  __in      LPCTSTR lpServiceName,
  __in      LPHANDLER_FUNCTION_EX lpHandlerProc,
  __in_opt  LPVOID lpContext
);

As lpContext was missing from the older RegisterServiceCtrlHandler it was not uncommon to create a bit of self modifying code that enabled lpHandlerProc to handle the 'this' pointer. In short create a function for each object conforming to the expected signature, that pushed the 'this' pointer on the stack.

This worked fine for me until XP service pack 2, Windows Server 2003 Sp1, and then a lot of services had to be fixed - so even if you do find a scenario where it solves a real problem and works, there is always the possibility that it will stop working in the future.

Best regards
Espen Harlinn