In my previous post, Kinect – Create Buttons, I’ve showed one approach how to create Kinect Buttons for Windows. Over the next posts, I’ll show more ways to accomplish that by moving windows Cursor based on Kinect Skeleton Right Hand Position.
Why To Create Kinect Button?
Why not use windows cursor and create brilliant hand or head () movement to simulate Click, so we don’t need to create designated Kinect Buttons.
Answers
- You had to stand at least 1 meter from the computer screen and even so I’m young I can’t see very well from that distance…
- Kinect precision is still not perfect and it’s hard to hit a 48x48 button from that distance.
- Even if you increase desktop resolution, it’s not easy and not practical to work with both hands to perform Click.
- But, I’m going to do that anyway! Magnifying glass, Cool Head or Hand movements, etc.
In this post, I’ll take those Kinect Button (working on Timer) implement on a simple Calculator
I’ve built and hook the Windows mouse to the Skeleton movement.
The only major issue with that is to adjust the Skeleton Position to your Screen Resolution, as you know the Skeleton coordinates are expressed in meters and comes from 640x480 screen, so I’ve create new class called Positions
that will help me to do it.
The AdjustToScreen
method gets the Joint (and based on the current screen resolution) and another gets the Joint and specific screen width and height.
public static class Positions
{
private const float SkeletonMaxX = 0.6f;
private const float SkeletonMaxY = 0.4f;
private static float Adjust(int primaryScreenResolution,
float maxJointPosition, float jointPosition)
{
var value = (((((float)primaryScreenResolution) / maxJointPosition) / 2f) *
jointPosition)
+ (primaryScreenResolution / 2);
if (value > primaryScreenResolution || value < 0f) return 0f;
return value;
}
public static Vector AdjustToScreen(Joint joint)
{
var newVector = new Vector
{
X = Adjust((int)SystemParameters.PrimaryScreenWidth,
SkeletonMaxX, joint.Position.X),
Y = Adjust((int)SystemParameters.PrimaryScreenHeight,
SkeletonMaxY, -joint.Position.Y),
Z = joint.Position.Z,
W = joint.Position.W
};
return newVector;
}
public static Vector AdjustToScreen(Joint joint, int screenWidth, int screenHeight)
{
var newVector = new Vector
{
X = Adjust(screenWidth, SkeletonMaxX, joint.Position.X),
Y = Adjust(screenHeight, SkeletonMaxY, -joint.Position.Y),
Z = joint.Position.Z,
W = joint.Position.W
};
return newVector;
}
}
Now I’ve created another helper class called – NativeMethods
to move the set the Cursor Position based on Skeleton Joint position.
public static class NativeMethods
{
public partial class MouseOperations
{
[DllImport("user32.dll")]
public static extern bool SetCursorPos(int x, int y);
[DllImport("user32.dll")]
public static extern bool GetCursorPos(out Point pt);
}
}
In the SkeletonFrameReady
I’ve added another quality check, also check the Joint TrakingState
and not just the Data TrakingState
, also make sure the joint position quality (W
) is high enough before moving the mouse.
The quality check will prevent that mouse from jumping around.
void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
{
if (data.TrackingState != SkeletonTrackingState.Tracked) continue;
foreach (Joint joint in data.Joints)
{
switch (joint.ID)
{
case JointID.HandRight:
if (joint.Position.W < 0.6f || joint.TrackingState !=
JointTrackingState.Tracked) return;
var newPos = Positions.AdjustToScreen(joint);
if (newPos.X == 0f || newPos.Y == 0f) return;
NativeMethods.MouseOperations.SetCursorPos(
Convert.ToInt32(newPos.X), Convert.ToInt32(newPos.Y));
break;
}
}
}
}
Make sure you enable TransformSmooth
, you don’t want the mouse jumping around. (Kinect–How to Apply Smooths Frame Display Using TransformSmoothParameters).
_kinectNui.SkeletonEngine.TransformSmooth = true;
var parameters = new TransformSmoothParameters
{
Smoothing = 1.0f,
Correction = 0.1f,
Prediction = 0.1f,
JitterRadius = 0.05f,
MaxDeviationRadius = 0.05f
};
_kinectNui.SkeletonEngine.SmoothParameters = parameters;
Live Demo
This project includes a Common class with this post main methods, from Moving and mouse and adjusting Kinect Skeleton to desktop resolution.
Enjoy!