Kinect – Calculator – Adjust Skeleton Movements To Mouse

In my previous post, Kinect – Create Buttons, I’ve showed one approach how to create Kinect Buttons for Windows. Over the next posts, I’ll show more ways to accomplish that by moving windows Cursor based on Kinect Skeleton Right Hand Position.

Why To Create Kinect Button?

Why not use windows cursor and create brilliant hand or head () movement to simulate Click, so we don’t need to create designated Kinect Buttons.

Answers

1. You had to stand at least 1 meter from the computer screen and even so I’m young I can’t see very well from that distance…
2. Kinect precision is still not perfect and it’s hard to hit a 48x48 button from that distance.
3. Even if you increase desktop resolution, it’s not easy and not practical to work with both hands to perform Click.
4. But, I’m going to do that anyway! Magnifying glass, Cool Head or Hand movements, etc.

In this post, I’ll take those Kinect Button (working on Timer) implement on a simple Calculator I’ve built and hook the Windows mouse to the Skeleton movement.

The only major issue with that is to adjust the Skeleton Position to your Screen Resolution, as you know the Skeleton coordinates are expressed in meters and comes from 640x480 screen, so I’ve create new class called Positions that will help me to do it.

The AdjustToScreen method gets the Joint (and based on the current screen resolution) and another gets the Joint and specific screen width and height.

public static class Positions
{
    //For - Resolution640x480
    private const float SkeletonMaxX = 0.6f;

    private const float SkeletonMaxY = 0.4f;
 

    private static float Adjust(int primaryScreenResolution, 
        float maxJointPosition, float jointPosition)
    {
        var value = (((((float)primaryScreenResolution) / maxJointPosition) / 2f) * 
            jointPosition) 
            + (primaryScreenResolution / 2);
            
        if (value > primaryScreenResolution || value < 0f) return 0f;
 
        return value;
    } 

    /// <summary>
    /// Get the current Joint position and Adjust the Skeleton 
    /// joint position to the current Screen resolution.
    /// </summary>
    /// <param name="joint">Joint to Adjust</param>

    /// <returns></returns>
    public static Vector AdjustToScreen(Joint joint)

    {
        var newVector = new Vector
        {
            X = Adjust((int)SystemParameters.PrimaryScreenWidth, 
            SkeletonMaxX, joint.Position.X),

            Y = Adjust((int)SystemParameters.PrimaryScreenHeight, 
            SkeletonMaxY, -joint.Position.Y),
            Z = joint.Position.Z,

            W = joint.Position.W
        };
 
        return newVector;
    }
    /// <summary>
    /// Get the current Joint position and Adjust the Skeleton joint
    /// position to a specific Screen Size.
    /// </summary>

    /// <param name="joint">Joint to Adjust</param>
    /// <param name="screenWidth">Screen Width</param>

    /// <param name="screenHeight">Screen Height</param>
    /// <returns></returns>

    public static Vector AdjustToScreen(Joint joint, int screenWidth, int screenHeight)
    {
        var newVector = new Vector
        {
            X = Adjust(screenWidth, SkeletonMaxX, joint.Position.X),
            Y = Adjust(screenHeight, SkeletonMaxY, -joint.Position.Y),

            Z = joint.Position.Z,
            W = joint.Position.W
        }; 

        return newVector;
    }
}

Now I’ve created another helper class called – NativeMethods to move the set the Cursor  Position based on Skeleton Joint position.

public static class NativeMethods
{
    public partial class MouseOperations
    {
        [DllImport("user32.dll")]
        public static extern bool SetCursorPos(int x, int y);
 
        [DllImport("user32.dll")]

        public static extern bool GetCursorPos(out Point pt);
    }
}

In the SkeletonFrameReady I’ve added another quality check, also check the Joint TrakingState and not just the Data TrakingState, also make sure the joint position quality (W) is high enough before moving the mouse.

The quality check will prevent that mouse from jumping around.

void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
    {
        //Tracked that defines whether a skeleton is 'tracked' or not. 
        //The untracked skeletons only give their position. 
        if (data.TrackingState != SkeletonTrackingState.Tracked) continue;
 
        //Each joint has a Position property that is defined by a Vector4: (x, y, z, w). 

        //The first three attributes define the position in camera space.
        //The last attribute (w)
        //gives the quality level (between 0 and 1) of the 

        foreach (Joint joint in data.Joints)
        {
            switch (joint.ID)
            {
                case JointID.HandRight:
                    if (joint.Position.W < 0.6f || joint.TrackingState != 
                        JointTrackingState.Tracked) return;// Quality check 

                    //Based on Skeleton Right Hand Position adjust the location
                    //to the screen

                    var newPos = Positions.AdjustToScreen(joint);
                    if (newPos.X == 0f || newPos.Y == 0f) return;

                    NativeMethods.MouseOperations.SetCursorPos(
                        Convert.ToInt32(newPos.X), Convert.ToInt32(newPos.Y));

                    break;
            }
        }
    }
}

Make sure you enable TransformSmooth, you don’t want the mouse jumping around. (Kinect–How to Apply Smooths Frame Display Using TransformSmoothParameters).

_kinectNui.SkeletonEngine.TransformSmooth = true;
var parameters = new TransformSmoothParameters
{
    Smoothing = 1.0f,

    Correction = 0.1f,
    Prediction = 0.1f,

    JitterRadius = 0.05f,
    MaxDeviationRadius = 0.05f
};
_kinectNui.SkeletonEngine.SmoothParameters = parameters;

Live Demo

Demo Project

This project includes a Common class with this post main methods, from Moving and mouse and adjusting Kinect Skeleton to desktop resolution.

Enjoy!