Click here to Skip to main content
15,867,686 members
Articles / Programming Languages / C#

Using Cognitive Services to Identify Persons

Rate me:
Please Sign up or sign in to vote.
5.00/5 (2 votes)
4 Jun 2018CPOL11 min read 11.2K   284   15   3
Use Face API to identify faces

Introduction

In my previous article, Using Cognitive Services to find your Game of Thrones look-alike, I have shown how you can use Face API from Azure Cognitive Services to find people that look alike. In that article, I built a demo app to identify somebody's most look-alike character from the Game of Thrones show, although the app could be used for other similar purposes. However, the Face API service has more algorithms than the face detection and similarity identification. With Face API, you can also identify people from a group. This article is a follow up in which I will show how to do that and will build yet another demo app to identify people of the houses from the same Game of Thrones show.

In order to see how to get started with Face API, please refer to the instructions from the previous article.

Understanding the API

The Face API reference documentation is available here. In there, you can find details about each API call, such as functionality, arguments, results, errors, and more.

In order to be able to identify people, you must do the following:

  • Create a person group. There are two types or groups: regular groups and large groups. You can have different numbers of groups depending on your subscription. For a free-tier, you can have a maximum of 1,000 person groups, each having up to 1,000 persons, but you cannot have more than 1,000 persons in all the person groups. For a S0-tier, you can have a maximum of a 1,000,000 person groups, each having up to 10,000 persons, but not more than 100,000,000 persons in all groups. If you need to have more than 10,000 persons in a group, you must use large person groups, which can accommodate up to 1,000,000 persons with the S0-tier. In this case, you can have up to 1,000,000 large person groups, but not more than 1,000,000,000 persons in all large person groups. With the free-tier, you can have 1,000 large person groups but not more than 1,000 persons in all the large person groups.
  • Create persons in the person group (with the limitations described above).
  • Add faces to each person in the group. For each person, whether you use regular or large groups, you can add up to 248 faces.
  • Train the person group in order to be able to identify persons. This is an asynchronous operation and the training time depends on the number of persons and their faces. This operation must complete before identification can proceed. This operation has to be executed again if you add more persons to the group or more faces to existing persons after the training completed.
  • Identify the person using the Face – Identify algorithm. You must specify a person group and the identifier of a face detected by Face – Detect.

The APIs you must use in order to manage person groups, persons and faces are the following:

  • Create a person group: you need to make a PUT HTTP request to [endpoint]/persongroups/{personGroupId}, where endpoint is the one you copied from the overview panel and personGroupId is a required parameter representing the identifier of the group. It's maximum length is 64 characters; the valid characters include numbers, English letters in lower case only, '-' and '_'. This call creates an empty group to which you can then add persons.
  • Create a person in a group: you need to make a POST HTTP request to [endpoint]/persongroups/{personGroupId}/persons, where personGroupId is the identifier of the group. In the body of the request, you can specify a display name for the person (maximum length is 128 characters) and optional user-provided data attached to the person (maximum length is 16KB), both of these being strings. If this call is successful, you get back a person identifier.
  • Add faces to a person: you need to make a POST HTTP request to [endpoint]/persongroups/{personGroupId}/persons/{personId}/persistedFaces[?userData][&targetFace], where personGroupId is the identifier of the group and personId is the identifier of the person. userData is optional, user-provided data about the target face (maximum length is 1KB). targetFace is a value that must indicate the area of the face if, and only if, the specified picture has more than one face; in this case, if this parameter is missing, the call fails. The actual image containing the face can be passed in two ways: as an URL in a JSON object, using the application/json as content-type, or as binary data, using application/octect-stream for the content-type.
  • Train the person group: make a POST HTTP request to [endpoint]/persongroups/{personGroupId}/train to start this asynchronous operation. This operation may take various times to complete, depending on the number of persons in the group and their faces. You can check the status of the operation with a GET HTTP request to [endpoint]/persongroups/{personGroupId}/training. The return JSON object contains a property called status that can have one of the following values: notstarted, running, succeeded, failed. You can only proceed with person identification after the training completed successfully.

There are more APIs than the ones mentioned above. They allow you to not only create, but also update, delete or retrieve the person groups, persons or their faces. In the application provided with this article, all of these functionalities are implemented. You can find them in the FaceApi namespace.

After building the person group, adding persons and their faces and training the group we can actually try to identify a person from an image. To do this, we must call, in this order, the following APIs:

  • Detect: is a POST HTTP request to [endpoint]/detect[?returnFaceId][&returnFaceLandmarks][&returnFaceAttributes]. The image can be passed either as an URL or as a binary stream, just as in the case of the API for adding a face to a person. When the call is successful, the result contains an array of faces, and for each face an ID, rectangle, landmarks, and attributes. What is necessary for the next call is the ID. This identifier is stored on the server for 24 hours before it expires.
  • Identify: is a POST HTTP request to [endpoint]/identify. There are several parameters that you must provide in a JSON object: the identifier of the face as returned by Detect (within an array that can contain up to 10 elements) and the identifier of the person group or the large person group. Optionally, you can specify the maximum number of candidates returned (valid numbers are 1 to 100, with 10 being the default) and the confidence treshhold (that can be between 0 and 1). If you provide more than one face, each face is identified independently. Keep in mind that identification works well for frontal and near-frontal faces.

As in the case of face lists, adding faces to a person in a person group does not retain the actual image on the server, only information about the face. If you need to display the images in your application, you need to keep them in a place where they can later be retrieved when needed. This is the case with the demo application, as we will see later on.

In all these APIs, you need to pass the application key in the Ocp-Apim-Subscription-Key custom header.

Consuming the APIs from C#

In this section, I will show how you can easily consume the aforementioned APIs from C#. The following code can be found in the FaceApiUtils class in the demo application.

  • Creating a person group, with a user-defined identifier, name and description. If the call fails, this function throws an exception.
    C#
    class PersonGroupCreateRequest
    {
       public string Name { get; set; }
       public string UserData { get; set; }
    }
    
    public static async Task<bool> CreatePersonGroup
    (string personGroupId, string name, string description)
    {
       using (var client = new HttpClient())
       {
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/persongroups/{personGroupId}";
    
          var body = new PersonGroupCreateRequest()
          {
             Name = name,
             UserData = description
          };
          var bodyText = JsonConvert.SerializeObject(body);
    
          var httpContent = new StringContent(bodyText, Encoding.UTF8, "application/json");
    
          var response = await client.PutAsync(uri, httpContent);
          if (!response.IsSuccessStatusCode)
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = 
                 JsonConvert.DeserializeObject<FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                       (errorResponse.Error.Code, errorResponse.Error.Message);
          }
    
          return response.IsSuccessStatusCode;
       }
    }
  • Creating a person in a group, specified by its identifier; the person has a name and an optional description. If the call is successful, the function returns the identifier of the person; otherwise it throws an exception.
    C#
    class PersonCreateRequest
    {
       public string Name { get; set; }
       public string UserData { get; set; }
    }
    
    class PersonCreateResponse
    {
       public string PersonId { get; set; }
    }
    
    public static async Task<string> CreatePersonInGroup
           (string personGroupId, string personName, string personDescription)
    {
       using (var client = new HttpClient())
       {
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/persongroups/{personGroupId}/persons";
    
          var body = new PersonCreateRequest()
          {
             Name = personName,
             UserData = personDescription
          };
          var bodyText = JsonConvert.SerializeObject(body);
    
          var httpContent = new StringContent(bodyText, Encoding.UTF8, "application/json");
    
          var response = await client.PostAsync(uri, httpContent);
          if (response.IsSuccessStatusCode)
          {
             var content = await response.Content.ReadAsStringAsync();
             var result = JsonConvert.DeserializeObject<PersonCreateResponse>(content);
             return result.PersonId;
          }
          else
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = 
                 JsonConvert.DeserializeObject<FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                   (errorResponse.Error.Code, errorResponse.Error.Message);
          }
       }
    }
  • Adding a face to a person in a group; both the group and the person are specified by their identifier. The face is identified from an image uploaded as a binary stream. If the call is successful, the function returns the persisted ID of the face; otherwise it throws an exception.
    C#
    public static async Task<string> AddFaceToPerson
    (string personGroupId, string personId, byte[] image)
    {
       using (var client = new HttpClient())
       {
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/persongroups/
                    {personGroupId}/persons/{personId}/persistedFaces";
    
          var content = new ByteArrayContent(image);
          content.Headers.ContentType = 
             new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
          var response = await client.PostAsync(uri, content);
          if (response.IsSuccessStatusCode)
          {
             var responseBody = await response.Content.ReadAsStringAsync();
             var result = JsonConvert.DeserializeObject<FaceAddResponse>(responseBody);
             return result.PersistedFaceId;
          }
          else
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = 
                 JsonConvert.DeserializeObject<FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                   (errorResponse.Error.Code, errorResponse.Error.Message);
          }
       }
    }
  • Starting the training of the group after adding all the persons and their faces. The group is specified by its identifier. If the function succeeded, it returns true to indicate the successful start of the operation. If the function failed, it throws an exception.
    C#
    public static async Task<bool> TrainPersonGroup(string personGroupId)
    {
       using (var client = new HttpClient())
       {
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/persongroups/{personGroupId}/train";
    
          var httpContent = 
                 new StringContent(string.Empty, Encoding.UTF8, "application/json");
    
          var response = await client.PostAsync(uri, httpContent);
          if (!response.IsSuccessStatusCode)
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = 
                    JsonConvert.DeserializeObject<FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                   (errorResponse.Error.Code, errorResponse.Error.Message);
          }
    
          return response.IsSuccessStatusCode;
       }
    }
  • Retrieving the status of the training operation. The group is specified by its identifier. If the operation succeeded, it returns a value of type PersonGroupTrainingStatus; this is an enumeration containing all the possible status value. If the call failed, this function throws an exception.
    C#
    enum PersonGroupTrainingStatus
    {
       notstarted,
       running,
       succeeded,
       failed
    }
    
    class PersonGroupTrainingStatusResponse
    {
       public string Status { get; set; }
       public string CreatedDateTime { get; set; }
       public string LastActionDateTime { get; set; }
       public string Message { get; set; }
    
       public DateTime GetCreatedDateTime()
       {
          DateTime.TryParse(CreatedDateTime, out DateTime dt);
          return dt;
       }
    
       public DateTime GetLastActionDateTime()
       {
          DateTime.TryParse(LastActionDateTime, out DateTime dt);
          return dt;
       }
    
       public PersonGroupTrainingStatus GetPersonGroupTrainingStatus()
       {
          Enum.TryParse(Status, true, out PersonGroupTrainingStatus status);
          return status;
       }
    }
    
    public static async Task<PersonGroupTrainingStatus> 
           GetPersonGroupTrainingStatus(string personGroupId)
    {
       using (var client = new HttpClient())
       {
    
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/persongroups/{personGroupId}/training";
    
          var response = await client.GetAsync(uri);
          if (response.IsSuccessStatusCode)
          {
             var responseBody = await response.Content.ReadAsStringAsync();
             var result = JsonConvert.DeserializeObject
                          <PersonGroupTrainingStatusResponse>(responseBody);
             return result.GetPersonGroupTrainingStatus();
          }
          else
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = 
                 JsonConvert.DeserializeObject<FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                   (errorResponse.Error.Code, errorResponse.Error.Message);
          }
       }
    }

The following helper types were used in the functions above for handling errors:

C#
class FaceApiError
{
   public string Code { get; set; }
   public string Message { get; set; }
}

class FaceApiErrorResponse
{
   public FaceApiError Error { get; set; }
}   

class FaceApiException : Exception
{
  public string Code { get; private set; }

  public FaceApiException(string code, string message) : base(message)
  {
     Code = code;
  }
}

The other two face API calls to implement are Detect and Identify. They are shown below:

  • The Detect function takes an image as binary content and sends it to the server. When successful, it returns back a list of detected faces. There is various information for each face, but the only one that is necessary is the temporary face identifier.
    C#
    class FaceDetectResponse
    {
      public string FaceId { get; set; }
    
      public Rectangle FaceRectangle { get; set; }
    
      public FaceLandmarks FaceLandmarks { get; set; }
    
      public FaceAttributes FaceAttributes { get; set; }
    }
    
    public static async Task<List<FaceDetectResponse>> DetectFace(byte[] image)
    {
       using (var client = new HttpClient())
       {
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/detect";
          var content = new ByteArrayContent(image);
          content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
          var response = await client.PostAsync(uri, content);
          if (response.IsSuccessStatusCode)
          {
             var responseBody = await response.Content.ReadAsStringAsync();
             var result = JsonConvert.DeserializeObject
                          <List<FaceDetectResponse>>(responseBody);
             return result;
          }
          else
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = JsonConvert.DeserializeObject
                                 <FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                   (errorResponse.Error.Code, errorResponse.Error.Message);
          }
       }
    }
  • The Identify function uses the temporary face identifier return by Detect, the person group identifier and a number of maximum candidates that should be returned (which my default is one). When successful, it returns a list of identified persons, each containing the person ID and the confidence score.
    C#
    class FaceIdentifyRequest
    {
       public List<string> FaceIds { get; set;}
       public string PersonGroupId { get; set; }
       public int? MaxNumOfCandidatesReturned { get; set; }
       public double? ConfidenceThreshold { get; set; }
    }
    
    class FaceIdentifyCandidate
    {
       public string PersonId { get; set; }
       public double Confidence { get; set; }
    }
    
    class FaceIdentifyResponse
    {
       public string FaceId { get; set; }
       public List<FaceIdentifyCandidate> Candidates { get; set; }
    }
       
    public static async Task<List<FaceIdentifyResponse>> 
       Identify(string faceId, string personGroupId, int maxNumOfCandidatesReturned = 1)
    {
       using (var client = new HttpClient())
       {
          client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AppSettings.Key1);
    
          var uri = $"{AppSettings.Endpoint}/identify";
    
          var body = new FaceIdentifyRequest()
          {
             FaceIds = new List<string> { faceId },
             PersonGroupId = personGroupId,
             MaxNumOfCandidatesReturned = 
                 (1 <= maxNumOfCandidatesReturned && maxNumOfCandidatesReturned <= 5) ? 
                 maxNumOfCandidatesReturned : 1,
          };
          var bodyText = JsonConvert.SerializeObject(body);
    
          var httpContent = new StringContent(bodyText, Encoding.UTF8, "application/json");
    
          var response = await client.PostAsync(uri, httpContent);
          if (response.IsSuccessStatusCode)
          {
             var responseBody = await response.Content.ReadAsStringAsync();
             var result = JsonConvert.DeserializeObject<List<FaceIdentifyResponse>>
                          (responseBody);
             return result;
          }
          else
          {
             var errorText = await response.Content.ReadAsStringAsync();
             var errorResponse = 
                 JsonConvert.DeserializeObject<FaceApiErrorResponse>(errorText);
             throw new FaceApiException
                   (errorResponse.Error.Code, errorResponse.Error.Message);
          }
       }
    }

Note: In all these code samples, AppSettings.Key1 and AppSettings.Endpoint are variables whose value is read from the application config file and represent the key and endpoint of your Face API Azure resource.

Note: There are additional types not listed above, such as Rectangle, FaceLandmarks, or FaceAttributes. You can find all of them in the attached source code.

Building an Application

We will again build a simple WPF application to manage person groups, persons and their faces on one hand, and identify persons on the other hand. This will be very similar to the one created in the previous article. In the demo shown here, I have created a person group for each major house from the Game of Thrones show, and added several faces to each member of the house. However, if you run the application that is provided with the article, you have to use your own application key, build your own person groups, and add persons and faces to them, as these are not shared between subscriptions.

The WPF application has three main windows:

  • The start-up window that enables you to select an action: either manage the person groups or identify a person (within one of the available groups).

    Image 1

  • The person groups management window. Here, you can view existing person groups (the ones created by the demo app are standard groups), add new or delete existing. For each person group, you can view the existing persons, add new or delete existing. For each person, you can view the images used to add faces to the person, add more faces from a file or entire folder and delete existing faces. Because the server does not retain the images themselves, these are stored in a subfolder in the working directory. The name of the working directory is specified in the application config file. In this folder, there is one subfolder for each person group; the name of the folder is the person group identifier. For each person in the group there is a further sub-folder in the person group folder; its name is the identifier of the person. The images used for adding faces to each person are stored in the persons folder. The name of each image in this folder is the persistent face ID returned by the server.

    Image 2

  • The window for identifying persons. This allows you to select an image from disk and a target person group. When successfully detected, it displays the first image of the person from the person folder and the confidence score.

    Image 3

The app.config file contains several application settings: the endpoint for the Face API resource, the access keys, and the name of the folder where the group and person subfolders are created and the face images stored, as described above.

XML
<appSettings>
  <add key="Endpoint" value="https://westeurope.api.cognitive.microsoft.com/face/v1.0" />
  <add key="Key1" value="...(insert your key)..." />
  <add key="Key2" value="...(insert your key)..." />
  <add key="PersonGroupBaseFolder" value="persongroups" />
</appSettings>

To get the application running with reasonable results, you should have at least several faces (the maximum possible is 248). Also remember that identification works best for frontal and near-frontal faces.

Conclusions

The Face API algorithms can be used for multiple purposes, such as face detection (along with facial attributes, including age, gender, pose, smile, facial hair, and even emotion), finding similar faces, identifying faces, or automatically grouping faces based on their similarity. There are many applications where these functionalities could be incorporated, including identification systems, when you allow a person to access resources after proving their identify. However, you should be aware of the limitations. These algorithms do not make a difference between a live picture captured with a camera or a printed picture held in front of a camera. Should you want to build such a system, you could incorporate several detection mechanisms, such as both face and voice recognition (where you perhaps ask the user to repeat a random word). However, these are not secure methods and should be used with care.

History

  • 5th June, 2018: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Architect Visma Software
Romania Romania
Marius Bancila is the author of Modern C++ Programming Cookbook and The Modern C++ Challenge. He has been a Microsoft MVP since 2006, initially for VC++ and nowadays for Development technologies. He works as a system architect for Visma, a Norwegian-based company. He works with various technologies, both managed and unmanaged, for desktop, cloud, and mobile, mainly developing with VC++ and VC#. He keeps a blog at http://www.mariusbancila.ro/blog, focused on Windows programming. You can follow Marius on Twitter at @mariusbancila.

Comments and Discussions

 
QuestionPersonGroupBaseFolder Pin
sheikh mohd afiq ahlami mohd noor2-Mar-20 22:40
sheikh mohd afiq ahlami mohd noor2-Mar-20 22:40 
Questioncode? Pin
koo96-Jun-18 11:34
koo96-Jun-18 11:34 
AnswerRe: code? Pin
Marius Bancila7-Jun-18 19:32
professionalMarius Bancila7-Jun-18 19:32 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.