Kernel-Mode Debugging in a VM using Visual Studio 2012

Introduction

This tip is about how to setup Kernel-Mode debugging in a VM using Visual Studio 2012. This tip will be helpful for people new to driver development that wish to create, build, deploy, and debug a simple "HelloKernel" project.

Background

The setup process can be divided into three parts:

• Setting up the Virtual Machine
• Setting up Visual Studio 2012
• Setting up your HelloKernel project

Each part will be explained step by step.

How to Setup the Virtual Machine

The Virtual Machine setup was tested on VMWare 9.0 running Windows 7 x64 OS.

• Create a new VM, use default settings (tested on VmWare, Win7 x64)
• Add a serial port to the VM, follow these steps:
  • Right click VM -> Settings -> Add -> Serial Port -> Output to named pipe
  • Named pipe = \\.\pipe\com_2
  • First combo box = The end is the server
  • Second combo box = The other end is an application
  • Connect at power on = Checked
  • Finish
  • Yield CPU on poll = Checked
• In the VM, in an elevated command prompt window, enter the following commands:
  • bcdedit /debug on
  • bcdedit /dbgsettings serial debugport:2 baudrate:115200
• In Windows, search for "Manage advanced sharing settings":
  • Set "Turn on file and printer sharing"
  • Set "Turn off password protected sharing"
• In Windows, search for "Computer Management":
  • Local Users and Groups -> Users:
  • Administrator -> Right Click -> Properties -> Uncheck Account is disabled
  • Administrator -> Right Click -> Set Password -> Set your password
• Install WDK Test Target, the setup can be found on the host computer in the following path:
  "C:\Program Files (x86)\Windows Kits\8.1\Remote"
• Reboot the VM

How to Setup Visual Studio

• Install Visual Studio 2012 (or above) and WDK 8 (or above)
• Run Visual Studio as administrator
• On the main toolbar, press Driver-> Test -> Configure Computers -> Add New Computer:
  • Computer name = VM name
  • Select Provision computer and choose debugger settings
  • Next
  • Connection Type = Serial
  • Pipe = Checked
  • Reconnect = Checked
  • Port = \\.\pipe\com_2
  • Target Port = com2
  • Next -> Finish (Don’t worry if Status is Undetermined)
• Reboot the VM
• Create a snapshot in the VM called "Configured"
• You may be prompted for Username\Password, enter Administrator\YourPassword
• Important! Currently Visual Studio 2012 has a weird defect that causes the steps above to fail if your Windows username has spaces in it. The active username must have no spaces !

How to Setup Your HelloKernel Project

• Use this tutorial to create your HelloKernel project:
• In order to debug your project, do the following:
  • OPTION I
    • Press F6 (Important! A newly configured VM requires one successful build before debugging)
    • Press F5
    • When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
    • In the "Debugger Immediate Window" write the following commands:
    • bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
    • g (Tells the debugger to continue)
  • OPTION II
    • Open two instances of VS, one will be the Deployer and the second the Debugger
    • [In Debugger] On the main toolbar press Debug -> Attach to Process
    • [In Debugger] Select "Windows Kernel Mode Debugger" and attach to the VM
    • [In Debugger] When Visual Studio changes to DebugMode, on the main toolbar press Debug -> Break All
    • [In Debugger] In the "Debugger Immediate Window", write the following commands:
    • [In Debugger] bu HelloKernel!DriverEntry (Tells the debugger to break on DriverEntry)
    • [In Debugger] g (Tells the debugger to continue)
    • [In Deployer] Press F6
  • OPTION III
    • Setup VS as a Debugger, as done in OPTION II
    • Copy the package folder located at "HelloWorld\Win7Debug\HelloWorld Package" into the VM
    • In the VM, create a new file named install.bat with the following commands:
    • sc stop HelloWorld
    • "C:\DriverTest\devcon.exe" install "C:\HelloWorld Package\HelloWorld.inf" Root\HelloWorld
    • sc start HelloWorld
    • In the VM, run install.bat as administrator
    • If prompted, press "Install this driver software anyway"
    • In order to reinstall a newer version of the driver, just update the package folder and rerun install.bat
• Your code should break on DriverEntry, and you can start debugging

Tips

• Tip I - Be sure your solution configuration is valid (Win7 x86 Debug\Win7 x64 Release ...)
• Tip II - Redeploying a driver to the VM after it had one successful deployment may cause problems (Such as the VM wanting to reboot).
  One possible solution would be to set a valid DriverUnload function in your driver.
  The DriverUnload function is a member of the PDRIVER_OBJECT received at DriverEntry.
  Another solution would be to return to the "Configured" snapshot before each deployment.

Summary

I hope this tip will help people new to driver development in getting started quickly.

It may seem straight forward, but I must say I encountered many problems trying to setup my environment, with nearly no useful and complete help found on the web.