Using PDB files and symbols to debug your application

• Download source code - 259 KB

Introduction

Errare humanum est (Latin for "to err is human") is a well known proverb which is applicable in nearly every aspect of life. You probably won't be able to find a programmer who has never encountered an error when running a program and has searched for ages to find the cause of it. Today, where code is often widely spread or even outsourced to the Cloud (think of Windows Azure, for example), this task gets even more complicated. That is the reason why efforts were always made to find mechanics that simplify debugging. When working with C++, there is also another problem that makes error finding very hard. After compilation, there is nothing left that lets us map an error onto a particular line of source code or even to a certain function. All names are "lost".

With PDB files, Windows offers us a powerful tool to deal with the problem mentioned in the last paragraph as they preserve information about the executable after compilation. With the help of those files, you are able to recover the source code as it was before compilation from the bits and bytes at runtime. The whole API that covers the work with PDB files and symbols is extremely huge and powerful. It is my goal to provide you an introduction to every aspect of that API in this article. But due to the fact that it is so big, this may take some time, and if you are interested in the topic, it is a good idea to recheck if there are new versions of this article with more things explained.

Loading symbols

Loading symbols for a process is a very easy task. All you have to do is call SymInitialize with the handle of the process for which the symbols should be loaded. If you are debugging a foreign process, you should pass the handle of that process to the function and not the handle of the application that debugs the process, because you want to load the symbols of the process that gets debugged. As a process may have loaded additional modules (DLLs) which also could have symbols, it is important to see if the function also loads the symbols for those modules. In fact, you are free to choose if they should be loaded or not. The third parameter of SymInitialize is a boolean indicating if the process should be "invaded", which means that symbols for all loaded DLLs will be loaded too, or not.

Usually, SymInitialize is called right after the process has started, and SymCleanup is used to unload all symbols when the process finishes. When calling SymInitialize multiple times for the same process, it fails, giving ERROR_INVALID_PARAMETER with GetLastError. This is a bit confusing as you cannot really distinguish multiple calls from an invalid process handle. Nevertheless, in the code attached to this article, you will see that SymInitialize is called in the constructor of a class and can therefore be called multiple times, and ERROR_INVALID_PARAMETER is used like it only indicates multiple calls. When using GetCurrentProcess, this basically is safe enough.

Now, let's imagine you load the symbols for the process at the startup and perform some tasks in that process. This may load new DLLs in the address space of the application and the symbols for those modules may not yet be loaded as they were not present at startup. To make sure symbols for those modules are also loaded, the function SymRefreshModuleList can be used, which is pretty simple to understand. It is also possible to load the symbols for a specific module only using SymLoadModule64, but I recommend you to use it only if you are debugging a process and receive an event that a module was loaded, because in that case, you have all the information to represent the symbols for the module as it is currently loaded.

Obtaining and working with a symbol

After symbols are loaded, you may want to actually get a specific symbol. To obtain symbols, there are various functions which use different approaches:

• SymFromName - Searches a symbol by its name
• SymFromAddr - Searches a symbol by its address
• SymFromIndex - Searches a symbol by its index
• SymFromToken - Searches a symbol by its managed token

Depending on how you have received the symbol, the first things you probably want to know are the name of the symbol and/or its address. Both of them can be accessed in the SYMBOL_INFO structure that gets filled by the above functions. But there is an important thing with that structure: it has a variable length and you are responsible to allocate the space. This is because the name is stored in a variable length array and truncated to fit its size (indicated by a member). As this array is the last member of the struct, you can just allocate as much space as you need to store the whole struct plus the full name, and pass that pointer to the functions. Have a look at the following example:

char memory[sizeof(SYMBOL_INFO) + MAX_SYM_NAME];
PSYMBOL_INFO sym = reinterpret_cast<PSYMBOL_INFO>(memory);
sym->NameLen = MAX_SYM_NAME;
sym->SizeOfStruct = sizeof(SYMBOL_INFO);
SymFromName(GetCurrentProcess(), "myFunction", sym);
//...

// or using malloc:
PSYMBOL_INFO sym = reinterpret_cast<PSYMBOL_INFO>(
                      malloc(sizeof(SYMBOL_INFO) + MAX_SYM_NAME));
sym->NameLen = MAX_SYM_NAME;
sym->SizeOfStruct = sizeof(SYMBOL_INFO);
SymFromName(GetCurrentProcess(), "myFunction", sym);
//...
free(sym);

One of the most important methods is SymGetTypeInfo which gives a lot of information about a type. For example, if you have a symbol that represents a function, you could request the calling convention of its type. Now, most things you can query using SymGetTypeInfo are not from a symbol but from a type index. To get the type index, you either have to look inside the PDB or you can get it from a loaded symbol. The SYMBOL_INFO struct contains a member called TypeIndex which holds the index of the type. With SymGetTypeInfo, you can query a lot of different things with just one function. Therefore it needs a lot of documentation. The last parameter is depending on the third, and is a pointer to data that must match with the one requested. So for the calling convention, for example, it's a pointer to a DWORD, while for the name, it's a pointer to a WCHAR* where the name is stored into. Have a look at the documentation at MSDN where every single type of information is described.

After receiving a symbol using any of the above mentioned methods, you may want to enumerate all the symbols that act as children of that symbol. For example, in the symbol of a class, it would contain members, or in a function, the parameters and local variables. There are also functions that are very easy to use.

• SymEnumSymbols - Enumerates all symbols for the current context of a DLL using wildcards
• SymEnumSymbolsForAddr - Enumerates all symbols for an address

Inside the source code, you can find the following bit of code which describes how those functions work. The comments should explain it pretty well:

BOOL __stdcall EnumParamsCallback(PSYMBOL_INFO inf, ULONG size, PVOID param)
{
    // Transform the param back to the one it was before.
    std::vector<std::string>* params = (std::vector<std::string>*)param;
    if(inf == NULL)
        return true;

    // The flags contain various information on what type of symbol we have here
    // SYMFLAG_PARAMETER says that its a function parameter.
    if(inf->Flags & SYMFLAG_PARAMETER)
        params->push_back(inf->Name);
    return true;
}

bool SymbolFunctionEntry::GetFunctionParams(std::vector<std::string>& params)
{
    // We need to set up a stack frame that will be used by SymEnumSymbolsForAddr
    // Of course we dont have a real stack frame as we only want to query information
    // but its enough to set the instruction offset for SymEnumSymbols to work.
    IMAGEHLP_STACK_FRAME frame = { 0 };
    frame.InstructionOffset = m_symbol->Address;

    // With SymSetContext we can set the current context
    // in which symbols should be enumerated
    // and evaluated. The last parameter is reserved!
    // If the currently set context is the same
    // as we set the function will return false but
    // setting ERROR_SUCCESS because it actually didnt
    // fail but just didnt do anything.
    if(SymSetContext(GetCurrentProcess(), &frame, NULL) ==
                     FALSE && GetLastError() != ERROR_SUCCESS)
        return false;

    // The last parameter will be passed to every call to EnumParamsCallback.
    if(SymEnumSymbols(GetCurrentProcess(), 0, NULL,
                      EnumParamsCallback, (LPVOID)¶ms) == FALSE)
        return false;

    return true;
}

Stack Trace or taking runtime information

In this section, we will create a class that is able to build a trace of the calling thread's execution stack which will contain a list of all the functions that were called in a chain and the values and names of the parameters passed to the function. With it, the following code will result in what you can see in the screenshot below:

#include "StdAfx.h"

enum MyEnum
{
    Foo1,
    Foo2
};

void __stdcall foo(short a, float b, MyEnum c, bool* didSucceed)
{
    StackTrace trace;
    std::cout << trace;
}

int main(int argc, const char* argv[])
{
    SymInitialize(GetCurrentProcess(), NULL, TRUE);
    bool noError = true;
    foo(500, 32.4f, Foo1, &noError);
    std::cin.get();
    SymCleanup(GetCurrentProcess());
}

The stack walk

In order to achieve that, there are basically two critical points:

• Getting a list of addresses which represent the functions in the chain
• Getting the type and the values of the parameters

While the first one is not that hard, the second task will need a lot more work. To obtain a list of the functions in the current thread's execution context, there is a simple function which is called StackWalk64. All you need to pass to that function is information about the function entry where it should start to walk back. You may now ask how we can get information about the current function call because that is also a part of what we actually like to get from the system, and it is correct to ask that because it's the trickiest point in the whole process of enumerating the stack trace. The StackWalk64 function uses a STACKFRAME64 structure to get the information about the current frame, and will return the information about the next frame in such a structure. That is pretty cool because once we have set up the initial frame, we don't need to care about creating the information about the next frame to pass it to the function because the function already returns the data for the next frame it needs to search further.

With the STACKFRAME64 structure, we get into a region where our code is depending on the architecture of the processor a lot because it uses some of the processor's registers and interprets them. Because on x64 the meaning of the registers are particularly different from the ones in x86, the code needs to be changed if you are running on a different platform. In general, the function needs to have the frame pointer, the stack pointer, and the instruction offset of the current function. What registers are used for that depends on the platform used. I will always rely on an x86 architecture, but just have a look at MSDN to find the values for x64. In x86, the instruction counter is stored inside the eip register (which cannot be accessed directly), the frame pointer is stored in the ebp register, and the stack pointer in the esp register. If we create a trace for another process, these registers can easily be obtained using the GetThreadContext function (remember to suspend the thread first). But if we create a trace for the current thread (like in the example source), this task gets a bit trickier. The easiest way is to create some inline assembly code:

CONTEXT ctx;
        memset(&ctx, 0, sizeof(CONTEXT));
        ctx.ContextFlags = CONTEXT_FULL;
        __asm
        {
                        call eipDummy
eipDummy:        pop eax
                        mov ctx.Eip,        eax
                        mov ctx.Esp,        esp
                        mov ctx.Ebp,        ebp
        }

While esp and ebp are actually self explaining, eip may be a bit confusing. As I said before, the register eip will throw an error when accessed. But the trick is the following: When the processor sees the call mnemonic, it will first push the address where the function should return to on the stack (which is the instruction directly after call) and then jumps to the place indicated by the first argument (eipDummy). So basically, the address of eipDummy will be pushed on the stack and also executed. All we need to do now is pop it back from the stack into a register and store that as eip. Like that, we will have all the information needed to initialize the first frame.

STACKFRAME64 frFirst;
frFirst.AddrPC.Offset = ctx.Eip;
frFirst.AddrPC.Mode = AddrModeFlat;
frFirst.AddrFrame.Offset = ctx.Ebp;
frFirst.AddrFrame.Mode = AddrModeFlat;
frFirst.AddrStack.Offset = ctx.Esp;
frFirst.AddrStack.Mode = AddrModeFlat;

The address mode indicates that we use absolute addresses without segmentation. We can already use STACKFRAME64 to create a frame if we like to have the constructor of StackTrace included in the trace. If we don't create a frame for it, the constructor won't be in the list as the first call to StackWalk64 will return the previous frame which is the function that called the constructor. In the source code provided, the constructor is included in the list. Now we are ready for the calls to StackWalk64. It's a pretty simple part:

while(StackWalk64(IMAGE_FILE_MACHINE_I386, GetCurrentProcess(), GetCurrentThread(), 
      &frFirst, &ctx, NULL, SymFunctionTableAccess64, SymGetModuleBase64, NULL))
{
    // ...
}

While the first few parameters should be pretty clear, the last four may be of interest. During the stack walk, the function uses different information. For example, it needs to read the process' memory (which need not actually belong to the calling process if you create a trace for another process). The sixth parameter can be a function pointer that should be called whenever memory should be read. If you set it to NULL, it will use ReadProcessMemory. The same applies to the other parameters except that they are for different purposes (accessing the function table, ...).

The stack frame

The StackFrame class doesn't really cover anything new. It's comparable to the SymbolFunctionEntry from the last paragraph. It's new that the calling convention is requested using SymGetTypeInfo but that's nothing special and the return type of the frame can be accessed using SymGetTypeInfo but with TI_GET_TYPE, because the return type is actually the type the function resolves to. The interesting part starts in the FunctionObject class. This class can load information about objects from a function. It includes local variables, parameters, and the return type. While for the last one only the type is interesting, the first two also have a name and a value. Obtaining the name is very easy because it's already in the symbol included. Getting a valid string for the type and the value is a bit more complex.

In the function FunctionObject::LoadType, everything starts with getting the tag of the symbol. The tag represents the first information about the type. While there are many different tags, there are three groups that are really different. The first one is SymTagBaseType. The basic types are all the simple types like int, float, char, double, and so on. These types do not have a special entry in the PDB because they should be clear. As always, SymGetTypeInfo gives us the information required: TI_GET_BASETYPE to get the basic type and TI_GET_LENGTH to obtain the number of bytes the type uses. The latter is needed to distinguish between the various integers (char, short, long, long long). While the base types are pretty easy to understand, they imply a lot of writing to translate the BasicType enum into readable text.

The second type is the SymTagPointerType. If that tag is returned, we have a pointer. To obtain the type the pointer is pointing to, once again SymGetTypeInfo is used using the TI_GET_TYPE request. The returned value is again an index that can be used with SymGetTypeInfo. So we can actually restart doing the above: get the tag for the new type and decide what to do depending on that tag. For multiple pointers like int***, the LoadPointerType function gets called three times until we finally reach the int but because it was called three times, we know that it is an int*** and not an int.

And the last group is everything that wasn't covered in the above. Those are complex types that have an entry in the PDB, and SymGetTypeInfo with TI_GET_SYMNAME returns a string with the type of the object.

While obtaining the value of a basic type is very simple, for complex types, it gets more difficult. For basic types, you just interpret the offset on the frame pointer as a pointer to the given type and take its value, that's all. The only complex type I have implemented so far is the enum type. In the source code, all the information is inside the FunctionObject::LoadEnumValue function. The function tries to match a value with the values defined in the enum. The values inside the enum are interpreted as children of the type, like members of a class or a struct. To obtain all the children of a type, once again, the SymGetTyeInfo does all the work. TI_GET_CHILDRENCOUNT returns the number of children a type has (if any), and TI_FINDCHILDREN returns type indices for each of the values defined in the enum. Because children of an enum are constant expressions, it's possible to request that value using TI_GET_VALUE. By looping through all the children and comparing the value with the value passed to the function, it's possible to determine if the enum defined a name for that value.

That's all we need to do to get a pretty nice stack trace.

Next things to come

• Interpreting user defined data types (classes, structs, ...)
• Using the symbol and source server to obtain sources and symbols of Windows DLLs
• Anything I forgot and what comes into my mind during the writing of the above ;)