Click here to Skip to main content
15,867,141 members
Articles / Programming Languages / Assembler

Accessing Windows API with Plain x64 Assembly Language

Rate me:
Please Sign up or sign in to vote.
4.97/5 (11 votes)
7 Oct 2016CPOL3 min read 32K   1K   14   5
This article shows how to access Windows API with plain x64 assembly programming language (MASM style). It shall also give an overview, how to apply some programming techniques like OOP and multithreading on a low level.

Image 1

Introduction

This tool is a simple 64 bit file manager for Windows featuring a 100% assembly language source code. The tool shall be a sample for using advanced techniques in assembly language environment, like native 64 bit, object orientation and unwind ability for stack frames.

Wait, Assembly Language? Isn't It Dead Already?

No, it's not. While programming languages are getting more and more high level, assembly language is still the base of each and every modern device, smart phone, tablet, desktop or server. If you understand assembly language, each other language will be just a new set of syntax elements and a couple of interesting new concepts.

There is probably no useful reason to do a large project in assembly, but every programmer should at least have an idea of what is going on under the hood.

How Will Assembly Language Be Able to Use Modern Techniques as OOP?

Most of the common programming paradigms are not a property of the language, but a way how to write source code. The language can give you support to avoid mistakes and make writing easily, however you can still do it on your own.

This project features a fully object orientated style, multi threaded working and all the Windows features for x64 programs.

How to Work with Assembly Language?

Visual Studio 2013 contains the MASM (Microsoft Macro Assembler) which will be able to assemble the source files out of the box. If you encounter any problems with the project, please don't hesitate to contact.

For better layout of the ASM sources, we use a VStudio plug in for formatting the source code. This will become a sub project in near future (hopefully). Currently, the plug in is not worth a public release.

Using the Code

Some interesting code snippets:

ASM
//
// x64 procedure stack frame
//

actSetProgress	PROC	FRAME
		LOCAL	hwndList:QWORD	; progress list
		LOCAL	xItem:LVITEM	; new list item
		LOCAL	txBuffer [128]:WORD	; command name
		LOCAL	xRange:PBRANGE	; progress bar range

		push		rbp
		.pushreg	rbp
		push    	r12
		.pushreg	r12
		push		r15
		.pushreg	r15
		mov		rbp, rsp
		.setframe	rbp, 0

		sub		rsp, 8 + 2 * 128 + sizeof LVITEM + sizeof PBRANGE
		sub		rsp, 48
		.allocstack	48 + 8 + 2 * 128 + sizeof LVITEM + sizeof PBRANGE
		.endprolog

aspDone:	mov		rax, 0

		add		rsp, 48
		add		rsp, 8 + 2 * 128 + sizeof LVITEM + sizeof PBRANGE

		pop		r15
		pop		r12
		pop		rbp
		ret		0

		align		4

Procedure starts with PROC FRAME, since MASM cannot declare parameters in x64 mode nothing follows.

Afterwards local variables are declared, with appropriate data types. Used registers should be pushed and popped manually and declared to be so with .pushreg keyword. Space for variables must be allocated manually too and also space for any call usage inside this procedure (sub rsp, 48 in this example). Stack usage is declared with .allocstack. Since all calls are register based, stack must be corrected on return and use ret 0. Finally, the header end with .endprolog.

You must care about alignment of stack to be 8 bytes. Code should be aligned on 4.

Calling a register based function:

ASM
//
// register based call with more than 4 parameters
// first four paramters in registers, others on stack
// but keep space for first four parameters anyway
//

	lea	rax, [r15.VIEW_PARAM.txFontName]
	mov	[rsp + 104], rax
	mov	dword ptr [rsp + 96], DEFAULT_PITCH OR FF_DONTCARE
	mov	dword ptr [rsp + 88], DEFAULT_QUALITY
	mov	dword ptr [rsp + 80], CLIP_DEFAULT_PRECIS
	mov	dword ptr [rsp + 72], OUT_DEFAULT_PRECIS
	mov	dword ptr [rsp + 64], DEFAULT_CHARSET
	mov	dword ptr [rsp + 56], FALSE
	mov	dword ptr [rsp + 48], FALSE
	mov	eax, dword ptr [r15.VIEW_PARAM.unItalic]
	mov	dword ptr [rsp + 40], eax
	mov	eax, dword ptr [r15.VIEW_PARAM.unWeight]
	mov	dword ptr [rsp + 32], eax
	mov	r9d, 0
	mov	r8d, 0
	mov	edx, 0
	mov	ecx, dword ptr [r15.VIEW_PARAM.unFontSize]
	call	CreateFont

Registers rcx, rdx, r8 and r9 hold first four parameters. If more are required, they are placed on the stack. Keep space for first 64 bit parameters, as the procedure can write parameters back on stack if required.

Doing Basic Object Orientation

Each object has to be a small allocated memory block. It contains a list of the methods it defines (vtable), which will be always the first member in block. If you share this table between objects, it represents the difference between class and object. If you change methods in list, it be some kind of overload or inheritance.

Generate a new object via handmade new operation:

ASM
buttonNew	PROC	FRAME

	...

	; -- allocate and set vtable --

	call		GetProcessHeap
	test		rax, rax
	jz		btnExit

	mov		r8, sizeof CLASS_BUTTON
	mov		edx, HEAP_ZERO_MEMORY
	mov		rcx, rax
	call		HeapAlloc
	test		rax, rax
	jz		btnExit

	lea		rcx, [rax.CLASS_BUTTON.xInterface]
	mov		[rax.CLASS_BUTTON.vtableThis], rcx

	; -- fill in method pointers --

	lea		rdx, btnInit
	mov		[rcx.CLASS_BUTTON_IFACE.pfnInit], rdx
	lea		rdx, btnLoadConfig
	mov		[rcx.CLASS_BUTTON_IFACE.pfnLoadConfig], rdx
	lea		rdx, btnSaveConfig
	mov		[rcx.CLASS_BUTTON_IFACE.pfnSaveConfig], rdx
	lea		rdx, btnGetRect
	mov		[rcx.CLASS_BUTTON_IFACE.pfnGetRect], rdx
	lea		rdx, btnRender
	mov		[rcx.CLASS_BUTTON_IFACE.pfnRender], rdx

Objects size is allocated at process heap. The table with method pointers resists inside object data. All pointers are written to the table and the table pointer is filled into first position.

Sample class definition:

ASM
CLASS_BUTTON_IFACE	struc
 pfnInit		dq	?	; init button
 pfnLoadConfig		dq	?	; load configuration
 pfnSaveConfig		dq	?	; save configuration
 pfnGetRect		dq	?	; get buttons rect
 pfnRender		dq	?	; render single button rectangle
CLASS_BUTTON_IFACE	ends

CLASS_BUTTON		struc
 vtableThis		dq	?	; objects methods
 pxApp			dq	?	; parent object
 unCmd			dq	?	; command to execute
 txText			dw	32 dup(?)	; text on button
 txParam		dw	DEF_PATH_LENGTH dup(?)	; parameters for command
 xParams		VIEW_PARAM		; view parameters
 unShortcut		dq	?	; keyboard shortcut
 unCol			dq	?	; horizontal position
 unRow			dq	?	; vertical position
 txSection		dw	64 dup(?)	; button configuration section
 xInterface		CLASS_BUTTON_IFACE		; button interface
CLASS_BUTTON		ends

If the class is defined this way and kept properly aligned to 8 bytes, it will be fully compatible to Visual Studio C++ classes, so you can even interact with a C++ object or call assembly methods from C++.

Calling an object method (MS style, aka STDMETHODCALL):

ASM
//
// calling a method on an object
//
	lea	r9, txTitle
	mov	r8, [r12.CLASS_APP.unLang]
	mov	rdx, IDS_ABOUT_TITLE
	mov	rcx, r12
	mov	rax, [r12.CLASS_APP.vtableThis]
	call	[rax.CLASS_APP_IFACE.pfnLoadString]

Object has to be in rcx register, rax holds the vtable pointer, and the call goes to wanted method inside vtable.

History

  • October 2016: First release

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
CEO digital performance
Germany Germany
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
PraiseMy Note of 5 Pin
Doom For Ever15-Oct-16 7:23
professionalDoom For Ever15-Oct-16 7:23 
QuestionWell done! Pin
jo201211-Oct-16 4:28
jo201211-Oct-16 4:28 
Compliments, dp - that is a cute project Thumbs Up | :thumbsup:

Minor point re "MASM cannot declare parameters in x64 mode": It can. Simple example:

WndProc proc uses rsi rdi rbx hWnd:HWND, uMsg:UINT, wParam:WPARAM, lParam:LPARAM
LOCAL ps:PAINTSTRUCT
  int 3
  mov rax, hWnd


Depending on your prolog macro, you'll see something like mov rax, qword ptr [rbp + 0x10] in the debugger. What ML64 has "forgotten" is things like invoke, .if / .else / .endif, .Repeat ... .Until etc. - you may use HJWasm instead, it's not crippled and a perfect MASM clone.

Re FASM & macros: So far I've not seen a convincing example of FASM macros. Google for MasmBasic to see what is possible with MASM Cool | :cool:

And of course, for the sake of demonstrating the use of assembler, dp is right not to use macros here. However, for a bigger project (say, 5,000+ lines), macros are essential to be productive.

Btw loading the project with VS community took ages, as usual with that behemoth, and an error message "Build: 0 succeeded, 3 failed". VS suggests to retarget the project to the latest version, and voilà, good news: "Retargeting End: 3 completed, 0 failed, 0 skipped". Then, when re-building the successfully re-targeted project, VS fills the output window with a loooong list of cryptic error messages. Honestly, there are far better alternatives for assembler; even \Masm32\qEditor.exe does a better job.

modified 11-Oct-16 10:48am.

QuestionWhy not FASM? Pin
Thornik10-Oct-16 15:44
Thornik10-Oct-16 15:44 
AnswerRe: Why not FASM? Pin
digital performance11-Oct-16 3:11
professionaldigital performance11-Oct-16 3:11 
GeneralRe: Why not FASM? Pin
Thornik11-Oct-16 10:59
Thornik11-Oct-16 10:59 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.