Control NAO Using Kinect

• Download source - 3.7 KB

Introduction

This article shows how to control NAOqi Robots like (NAO, Pepper, Romeo) using Kinect device using NAO-Kinect.

Background

My dream is to control robots. Part of it is to control robots using Kinect and here is how I made it.

I developed a program before about controlling Servo using Kinect and Arduino in CodeProject. Also Control Arm Robot using Kinect in YouTube link. Now I upgraded it when I got my own NAO robot. I hope you like it and it may be useful in your programs.

What this Project Does

1. Catches joints position of joints for the person who stands in front of Kinect
2. Displays it in numbers and graphics on Windows Forms Application
3. Passes the Kinect Joints values from Windows to NAOqi Robot environment (Python)
4. Applies the moves on NAOqi-based robot like: (NAO, Pepper or romeo)

Prerequisites

To implement this project, you have to have some software and hardware:

Hardware

1. Kinect (V2 Recommended)
2. Windows Computer with USB3 Port
3. NAOqi-based Robot (optional)

Software

1. Kinect SDK

   From Aldebaran Website login/Register and download:

2. NAOqi SDK: "pynaoqi-2.1.4.13.win32.exe" (Recommended version)
3. Choregraphe: "choregraphe-suite-2.1.4.13-win32-setup.exe"(Recommended version)

Using the Code

The code is divided into four parts and rules.

1. Main
   • Type: Windows Application
   • Rule: User Interface
   • Description: Project which displays Kinect Joints values and Robot data and interacts with user actions
2. dllKinectCatcher
   • Type: Windows DLL
   • Rule: Kinect - Windows Interface
   • Description: Project which catches Required Events and live data values from Kinect
3. DllNAO.netV2
   • Type: Windows DLL
   • Rule: NAOqi - Windows Interface
   • Description: Control NAOqi-Based Robots using C#, it is an improvement for the original NAO.NET project listed down
4. Baku.libqiDotNet
   • Type: Windows DLL
   • Rule: dotnet NAoqi Interface
   • Description: Unofficial .NET wrapper library for "qi Framework", the messaging library created by Aldebaran Robotics

1. Main Project

The main project is a Windows Forms Application. (Introduction form and Control Form)

It is just a simple Windows Forms Application with many textBoxes to hold the joints coordinates (X,Y,Z) for every one. Live data display.

Main Project Points of Interest

The best interesting point about that is how all text boxes update at the same time with no hanging... it uses threads and working background to do that .

Sample Code

This example for how to update the hand's (left and right) values Async using BeginInvoke:

//Right Hand
txtKinRHandX.BeginInvoke
  (new Action(() => txtKinRHandX.Text = e.hand.Get(utilities.Side.Right).x.ToString()));
txtKinRHandY.BeginInvoke
  (new Action(() => txtKinRHandY.Text = e.hand.Get(utilities.Side.Right).y.ToString()));
txtKinRHandZ.BeginInvoke
  (new Action(() => txtKinRHandZ.Text = e.hand.Get(utilities.Side.Right).z.ToString()));

//Left Hand
txtKinLHandX.BeginInvoke
  (new Action(() => txtKinLHandX.Text = e.hand.Get(utilities.Side.Left).x.ToString()));
txtKinLHandY.BeginInvoke
  (new Action(() => txtKinLHandY.Text = e.hand.Get(utilities.Side.Left).y.ToString()));
txtKinLHandZ.BeginInvoke
  (new Action(() => txtKinLHandZ.Text = e.hand.Get(utilities.Side.Left).z.ToString()));

2. dllKinect

dllKinect is a Windows DLL project which be allowed to catch the joints data from Kinect and convert it to suitable data to read.

Why DLL Kinect?

Microsoft provides Kinect SDK for control Kinect, but this SDK provides only WPF platform but not Windows Application.

To make it easy to catch data into Windows Forms application, I had to develop this application which means that you can cancel this DLL project if you will develop your application directly using WPF.

Example Code

Note: This code is just an example about how to catch the joints.

public  KinectCatcher.utilities.Position Get(KinectCatcher.utilities.Side hand)
{
    KinectCatcher.utilities.Position position = new utilities.Position();
    switch (hand)
    {
        case KinectCatcher.utilities.Side.Right:
            position = RightHandPosition;
           return position;
        case KinectCatcher.utilities.Side.Left:
           position = LeftHandPosition;
             return position;  
        default:
             return new utilities.Position();
    }
}

3. DllNAO.netV2

Second main part of the solution is NAO.NET Project, which allows you to catch and convert the joints data, from Windows to NAOqi movement data which can be applied on the robot.

In the past, I posted another article about how to execute Python from Windows application and as an example, I can control NAO robot using it, you can check it here: NAO.NET V1. But it cannot fit in this case because it is generic, slow and there are many other problems, so I had to make something which uses another technique like messaging, and developed another version named NAO.NETV2.

Main Code

This is the main code which explains how NAO.NETv2 works. It can control any NAOqi robot using windows, which is amazing and has enough to be controlled.

try
{
    if (!session.IsConnected)
    {
        result.ResultStatus = ACall.structAcallResult.Result.failure;
        result.dateTime = DateTime.Now;
        result.Message = "Not Connected";
        return result;
    }
    var nAOQiService = session.GetService(NAOqiParameters.Service.ToString("F"));
    switch (NAOqiParameters.Service)
    {
        case NAOqiServices.ALMemory:
            break;
        case NAOqiServices.ALTextToSpeech:
        
            result.NAOqiObject = 
                nAOQiService[NAOqiParameters.method].Call(NAOqiParameters.SingleParamValue);
            result.Message = "called ALTextToSpeech Successfully";
            
            break;
        #region Motion Service
        case NAOqiServices.ALMotion:
            
            QiList< QiString> names
                        = QiList.Create(NAOqiParameters.names);
            QiList< QiList<QiDouble>> angleLists = 
                QiList.Create< QiList< QiDouble>>(new QiList<QiDouble>[] 
                { QiList.Create(NAOqiParameters.AngleList[0]) });
            
            for ( int i = 1 ; i < NAOqiParameters.AngleList.Count(); i++ ) 
            {
                angleLists.QiValue.AddElement
                        (QiList.Create(NAOqiParameters.AngleList[i]).QiValue);
            }
            QiList< QiList<QiDouble>> timeLists = 
                QiList.Create< QiList<QiDouble>>(new QiList<QiDouble>[] 
                { QiList.Create(NAOqiParameters.timeLists[0]) });
            for ( int i = 1; i < NAOqiParameters.timeLists.Count() ; i++ )
            {
                timeLists.QiValue.AddElement
                     (QiList.Create(NAOqiParameters.timeLists[i]).QiValue);
            }
           QiBool isAbsolute;
            if (NAOqiParameters.isAbsolute)
            {
                isAbsolute = new QiBool(true);
            }
            else
            {
                isAbsolute = new QiBool(false);
            }
            result.NAOqiObject = nAOQiService["angleInterpolation"].Call
                         (names, angleLists, timeLists, isAbsolute);
            result.Message = "successfully called ALMotioin Service";
            
            break;
        #endregion Motion Service
        default:
            break;
    }
    
    result.dateTime = DateTime.Now;
 
    result.ResultStatus = ACall.structAcallResult.Result.Sucess;
    session.Close();
    session.Destroy();
    return result;
}

History

Of course, I want to have more control and integration of this project, I may update more code and control options later on gitHub as well. For more projects or help, you can contact me by leaving a comment below.