Bing Visual Search for Image Insights

Introduction

The Visual Search SDK supports Bing Image Search functionality and simplifies the details of sending a POST request for image insights.

Background

Image insights include related images, web pages that contain the image, or lists of merchants where the product shown in the image can be purchased. For more information, see the Microsoft documentation: Get insights about an image and the Visual Search API.

There are several ways to get image insights. The methods require sending an image or insights token from a previous image request to Bing for further information known as image insights. The Bing Search SDK economically manages details of the POST request, so you don't have to construct the POST body and metadata as required by the REST API.

Using the Code

The Visual Search Bing SDK is available in four languages:

• C#
• Node
• Java
• Python

The following code sample uses the C# SDK, which can be installed for Visual Studio development using a Nuget package.

Application Dependencies

To set up a console application using the Bing Visual Search SDK, browse to the Manage NuGet Packages option from the Solution Explorer in Visual Studio. Add the Microsoft.Azure.CognitiveServices.Search.VisualSearch package.

Installing the NuGet Web Search SDK package also installs dependencies, including:

• Microsoft.Rest.ClientRuntime
• Microsoft.Rest.ClientRuntime.Azure
• Newtonsoft.Json

For this quickstart, you can use a free trial subscription key or a paid subscription key.

Basic Code Scenario

The following code shows the basic scenario for image insights using binary image upload. The example instantiates the VisualSearchAPI client, POSTS an image file from local TestImages folder, then lists tags and actions from the results of the search.

Tags help users explore the subjects found in the image. For example, if the input image is of a recognizable celebrity, one of the tags could be the name of the person, another tag the area of expertise of the person in the image.

Actions identify various information about an image, such as shopping sources, related searches, or Web pages that include the image.

using Microsoft.Azure.CognitiveServices.Search.VisualSearch;
using Microsoft.Azure.CognitiveServices.Search.VisualSearch.Models;
using System;
using System.IO;
using System.Linq;

namespace CodeProjectVisualSrch
{
    class Program
    {
        static void Main(string[] args)
        {
            String subscriptionKey = "YOUR-VISUAL-SEARCH-KEY";
            try
            {
                var client = new VisualSearchAPI(new ApiKeyServiceClientCredentials(subscriptionKey));

                using (System.IO.FileStream stream = 
                   new FileStream(Path.Combine("TestImages", "image.jpg"), FileMode.Open))
                {
                    // The knowledgeRequest parameter is not required 
                    // if an image binary is passed in the request body
                    var visualSearchResults = client.Images.VisualSearchMethodAsync
                                          (image: stream, knowledgeRequest: (string)null).Result;
                    Console.WriteLine("\r\nVisual search request with binary of image");

                    if (visualSearchResults == null)
                    {
                        Console.WriteLine("No visual search result data.");
                    }
                    else
                    {
                        // List of tags
                        if (visualSearchResults.Tags.Count > 0)
                        {
                            var firstTagResult = visualSearchResults.Tags.First();
                            Console.WriteLine($"Visual search tag count: 
                                                {visualSearchResults.Tags.Count}");

                            // List of actions in first tag
                            if (firstTagResult.Actions.Count > 0)
                            {
                                var firstActionResult = firstTagResult.Actions.First();
                                Console.WriteLine($"First tag action count: 
                                                {firstTagResult.Actions.Count}");
                                Console.WriteLine($"First tag action type: 
                                                {firstActionResult.ActionType}");
                            }
                            else
                            {
                                Console.WriteLine("Couldn't find tag actions!");
                            }
                        }
                        else
                        {
                            Console.WriteLine("Couldn't find image tags!");
                        }
                    }
                }
            }

            catch (Exception ex)
            {
                Console.WriteLine("Encountered exception. " + ex.Message);
            }

            // These functions are described in subsequent headings of the article.
            VisualSearchImageBinaryWithCropArea(subscriptionKey);
            VisualSearchUrlWithFilters(subscriptionKey);
            VisualSearchInsightsTokenWithCropArea(subscriptionKey);

            Console.WriteLine("Any key to quit...");
            Console.ReadKey();
        }
    }
}

Crop Area of Image

You can also send a query for a section of an image by setting a crop area by coordinates, top left and bottom right. The coordinates are specified by percentage in decimal format: 0.0 – 1.0.

The following function sends an image binary in the body of the post request along with a crop area object. Then it prints out the number of tags, the number of actions, and the first action type.

public static void VisualSearchImageBinaryWithCropArea(string subscriptionKey)
{
    var client = new VisualSearchAPI(new ApiKeyServiceClientCredentials(subscriptionKey));

    try
    {
        using (FileStream stream = new FileStream
                 (Path.Combine("TestImages", "image.jpg"), FileMode.Open))
        {
            // The ImageInfo struct contains a crop area specifying a region 
            // to crop in the uploaded image
            CropArea CropArea = new CropArea(top: (float)0.1, 
                     bottom: (float)0.5, left: (float)0.1, right: (float)0.9);
            ImageInfo ImageInfo = new ImageInfo(cropArea: CropArea);
            VisualSearchRequest VisualSearchRequest = new VisualSearchRequest(imageInfo: ImageInfo);

            // The knowledgeRequest here holds additional information about the image, 
            // which is passed in binary form
            var visualSearchResults = client.Images.VisualSearchMethodAsync
                             (image: stream, knowledgeRequest: VisualSearchRequest).Result;
            Console.WriteLine("\r\nVisual search request with binary of image 
                             and knowledgeRequest of crop area");

            if (visualSearchResults == null)
            {
                Console.WriteLine("No visual search result data.");
            }
            else
            {
                // List of tags
                if (visualSearchResults.Tags.Count > 0)
                {
                    var firstTagResult = visualSearchResults.Tags.First();
                    Console.WriteLine($"Visual search tag count: {visualSearchResults.Tags.Count}");

                    // List of actions in first tag
                    if (firstTagResult.Actions.Count > 0)
                    {
                        var firstActionResult = firstTagResult.Actions.First();
                        Console.WriteLine($"First tag action count: {firstTagResult.Actions.Count}");
                        Console.WriteLine($"First tag action type: {firstActionResult.ActionType}");
                    }
                    else
                    {
                        Console.WriteLine("Couldn't find tag actions!");
                    }
                }
                else
                {
                    Console.WriteLine("Couldn't find image tags!");
                }
            }
        }
    }

    catch (Exception ex)
    {
        Console.WriteLine("Encountered exception. " + ex.Message);
    }
}

URL of Image

Instead of an image binary, you can send the URL of an image in an image info object. Optional filters in a knowledge request object can specify a domain to search.

The following function sends an image URL in the knowledgeRequest parameter, along with a filter: site:www.bing.com. The code prints the results including the number of tags, the number of actions, and the first actionType.

This example also prints out the imageInsightsToken that is uploaded with the results.

public static void VisualSearchUrlWithFilters(string subscriptionKey)
{
    var client = new VisualSearchAPI(new ApiKeyServiceClientCredentials(subscriptionKey));

    try
    {
        // The image can be specified via URL, in the ImageInfo object
        var ImageUrl = "https://images.unsplash.com/photo-1512546148165-e50d714a565a?w=600&q=80";
        ImageInfo ImageInfo = new ImageInfo(url: ImageUrl);

        // Optional filters inside the knowledgeRequest will restrict similar products 
        // and images to certain domains
        Filters Filters = new Filters(site: "www.bing.com");
        KnowledgeRequest KnowledgeRequest = new KnowledgeRequest(filters: Filters);

        // An image binary is not necessary here, as the image is specified via URL
        VisualSearchRequest VisualSearchRequest = new VisualSearchRequest
                           (imageInfo: ImageInfo, knowledgeRequest: KnowledgeRequest);
        var visualSearchResults = client.Images.VisualSearchMethodAsync
                           (knowledgeRequest: VisualSearchRequest).Result;
        Console.WriteLine("\r\nVisual search request with url of image and 
                            knowledgeRequest based on filters");

        if (visualSearchResults == null)
        {
            Console.WriteLine("No visual search result data.");
        }
        else
        {
            // Visual Search results
            if (visualSearchResults.Image?.ImageInsightsToken != null)
            {
                Console.WriteLine($"Uploaded image insights token: 
                              {visualSearchResults.Image.ImageInsightsToken}");
            }
            else
            {
                Console.WriteLine("Couldn't find image insights token!");
            }

            // List of tags
            if (visualSearchResults.Tags.Count > 0)
            {
                var firstTagResult = visualSearchResults.Tags.First();
                Console.WriteLine($"Visual search tag count: {visualSearchResults.Tags.Count}");

                // List of actions in first tag
                if (firstTagResult.Actions.Count > 0)
                {
                    var firstActionResult = firstTagResult.Actions.First();
                    Console.WriteLine($"First tag action count: {firstTagResult.Actions.Count}");
                    Console.WriteLine($"First tag action type: {firstActionResult.ActionType}");
                }
                else
                {
                    Console.WriteLine("Couldn't find tag actions!");
                }
            }
            else
            {
                Console.WriteLine("Couldn't find image tags!");
            }
        }
    }

    catch (Exception ex)
    {
        Console.WriteLine("Encountered exception. " + ex.Message);
    }
}

Insights Token with Crop Area

If you have an image insights token from a previous request using the Bing Image Search API, you can send the image insights token in the knowledgeRequest parameter instead of an image binary or URL.

The following uses an image insights token along with a crop area object to get results. Then it prints out the imageInsights token, the number of tags, the number of actions, and the first action type.

public static void VisualSearchInsightsTokenWithCropArea(string subscriptionKey)
{
    var client = new VisualSearchAPI(new ApiKeyServiceClientCredentials(subscriptionKey));

    try
    {
        // The image can be specified via an insights token, in the ImageInfo object
        var ImageInsightsToken = "bcid_113F29C079F18F385732D8046EC80145*ccid_oV/
        QcH95*mid_687689FAFA449B35BC11A1AE6CEAB6F9A9B53708*thid_R.113F29C079F18F385732D8046EC80145";

        // An optional crop area can be passed in to define a region of interest in the image
        CropArea CropArea = new CropArea(top: (float)0.1, bottom: (float)0.5, 
                            left: (float)0.1, right: (float)0.9);
        ImageInfo ImageInfo = new ImageInfo(imageInsightsToken: 
                              ImageInsightsToken, cropArea: CropArea);

        // An image binary is not necessary here, as the image is specified via insights token
        VisualSearchRequest VisualSearchRequest = new VisualSearchRequest(imageInfo: ImageInfo);
        var visualSearchResults = client.Images.VisualSearchMethodAsync
                                  (knowledgeRequest: VisualSearchRequest).Result;

        Console.WriteLine("\r\nVisual search request with 
                          imageInsightsToken and knowledgeRequest based on imageInfo");

        if (visualSearchResults == null)
        {
            Console.WriteLine("No visual search result data.");
        }
        else
        {
            // Visual Search results
            if (visualSearchResults.Image?.ImageInsightsToken != null)
            {
                Console.WriteLine($"Uploaded image insights token: 
                                  {visualSearchResults.Image.ImageInsightsToken}");
            }
            else
            {
                Console.WriteLine("Couldn't find image insights token!");
            }

            // List of tags
            if (visualSearchResults.Tags.Count > 0)
            {
                var firstTagResult = visualSearchResults.Tags.First();
                Console.WriteLine($"Visual search tag count: {visualSearchResults.Tags.Count}");

                // List of actions in first tag
                if (firstTagResult.Actions.Count > 0)
                {
                    var firstActionResult = firstTagResult.Actions.First();
                    Console.WriteLine($"First tag action count: {firstTagResult.Actions.Count}");
                    Console.WriteLine($"First tag action type: {firstActionResult.ActionType}");
                }
                else
                {
                    Console.WriteLine("Couldn't find tag actions!");
                }
            }
            else
            {
                Console.WriteLine("Couldn't find image tags!");
            }
        }
    }

    catch (Exception ex)
    {
        Console.WriteLine("Encountered exception. " + ex.Message);
    }
}

History

• Version 1.0