The Kinect Mouse Controller C#

• Download the DLL - 2.5 KB

The zip file provided with this article contains a DLL file KinectMouseController.DLL that could be used in all kinds of C# projects to control the cursor of Windows OS system from a Kinect device.

This schema explains how the provided DLL could be useful to send position (x,y) from the Kinect device to the Windows cursor:

Introduction

Controlling the cursor using the Kinect is a common question that I have noticed in many web sites, This feature could help a lot of people to make something more interactive, Designer, Architect, exploring maps, exploring 3D building ,... etc.

So in this quick description, I will explain how to add this feature to your project and also provide an application that could help people who aren't familiar with User32.dll and Kinect SDK.

Background

Having an experience with Kinect SDK isn't a must to benefit from this DLL,you should have some basics in coding a Kinect application to implement cursor controller in your Kinect project and use my code.

You should have a Kinect device, install the Kinect SDK and have hardware requirements to benefit from Kinect device in your computer.

Learn my first article about the Kinect for more details, and take a look at this.

What is User32 ?

User32 is one of the DLLs cached on system32 of Windows OS.

It's a Microsoft DLL that provides a lower-level control of Windows, menu, mouse, keyboard,... etc, it's based on many native methods; MOUSE_EVENT etc. that allows a control of a Windows user interface.

Using the Code

To use the DLL KinectMouseController in your project, you should first add a reference to this DLL in your project.

So after creating a project that will use the Kinect SDK, right click the reference folder of your project:

Choose the Browse tab, and select the KinectMouseController.DLL:

In your .cs file where you need to use the mouse controller, add a using KinectMouseController:

Now, you can use the following method:

SendMouseInput(int CursorX, int CursorY,int ScreenWidth, int ScreenHeight, bool isClicked)

In this method, we give the mouse position, the screen dimension and a boolean that represents a mouse click, true if down and false if up.

KinectMouseController.KinectMouseMethods.SendMouseInput(cursorX, cursorY, 
 (int)SystemInformation.PrimaryMonitorSize.Width, 
 (int)SystemInformation.PrimaryMonitorSize.Height, true); 

This code could be used in nui_SkeletonFrameReady() of Kinect SDK, in this code I'll use it to control the mouse using my right hand ,e.g :

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
       { 
            SkeletonFrame allSkeletons = e.SkeletonFrame;

            //get the first tracked skeleton
            SkeletonData skeleton = (from s in allSkeletons.Skeletons 
                                     where s.TrackingState == SkeletonTrackingState.Tracked
                                     select s).FirstOrDefault();
            //Test if skelton exist and tracked
            if (skeleton != null && skeleton.TrackingState == SkeletonTrackingState.Tracked)
            {
                SwG.Add(skeleton.Joints[JointID.HandRight].Position, nui.SkeletonEngine);

                // scale those Joints to the primary screen width and height 
                Joint scaledRight = skeleton.Joints[JointID.HandRight].ScaleTo((
                    int)SystemInformation.PrimaryMonitorSize.Width, 
                    (int)SystemInformation.PrimaryMonitorSize.Height, 
                    SkeletonMaxX, SkeletonMaxY);

                int cursorX = (int)scaledRight.Position.X;
                int cursorY = (int)scaledRight.Position.Y;
KinectMouseController.KinectMouseMethods.SendMouseInput
	(cursorX, cursorY, (int)SystemInformation.PrimaryMonitorSize.Width, 
	(int)SystemInformation.PrimaryMonitorSize.Height, false); 
          }

In the nui_SkeletonFrameReady(), we detect the first Skelton, after that we get the right hand portion of this skelton cursorX and cursorY, those coordinates are passed to the SendMouseInput() method of the KinectMouseController to take control of the mouse using the right hand of the first detected Skelton.

Points of Interest

I also wrote a related article:

• How-to: Benefit from Kinect.Toolbox and Coding4Fun on Kinect Programming

Note

This code is tested with Windows 7, Kinect SDK beta 1 and Visual Studio 2010.