Click here to Skip to main content
15,902,189 members
Articles / Artificial Intelligence / Deep Learning

AI Social Distancing Detector: Using OpenCV to Annotate Detected Objects

Rate me:
Please Sign up or sign in to vote.
2.90/5 (8 votes)
3 Dec 2020CPOL4 min read 5.6K   125   5  
In this article, we will learn how to add annotations to images.
Here we will learn how to use mouse callbacks in OpenCV to select various objects in images. This functionality is commonly used to prepare test and training datasets, and to indicate detected objects.

In the first article of this series, we learned how to use OpenCV to work with images and video sequences. We learned how to read, write and display images and videos from a webcam and video files.

In this article, we will learn how to annotate objects in images with OpenCV as shown in the image below. You typically need this during object detector training to indicate objects in the images in your training and test datasets.

OpenCV provides a mechanism that enables you to select shapes in images. Namely, you can wire the mouse callbacks with the windows you display with OpenCV (imshow or namedWindow methods). By doing so, you can detect when the user clicks the mouse buttons and where they put the pointer.

I will show you how to respond to mouse events with OpenCV and Python. You can find the companion source code here.

Common Module

To group the various constants I used in the code, I created the module:

import cv2 as opencv
LENA_FILE_PATH = '../Images/Lena.png'
# Colors (BGR)
GREEN = (0, 255, 0)
# Image labeling parameters

This module defines several constants:

  • LENA_FILE_PATH – Points to the Lena image location.
  • WINDOW_NAME – Stores the window name.
  • GREEN and LINE_THICKNESS – Defines the color (in a BGR format) and line thickness used to draw a rectangle.
  • FONT_FACE, FONT_SCALE, etc. – Parameters for displaying the object label.
  • TEXT_OFFSET – Defines the gap (in pixels) between the label and the rectangle.

Mouse Callback

To handle mouse interaction with the OpenCV window, you set the mouse callback. To do so, you first create the window using the namedWindow function:

import cv2 as opencv
import common

Then, invoke setMouseCallback as follows:

opencv.setMouseCallback(common.WINDOW_NAME, on_mouse_move)

The callback should have the following prototype:

def on_mouse_move(event, x, y, flags, param)

These are the five input parameters:

  • x, y – The coordinates of the mouse cursor.
  • flags – Flags indicating whether the mouse button is down (EVENT_FLAG_LBUTTON, EVENT_FLAG_RBUTTON, EVENT_FLAG_MBUTTON) or the user presses Ctrl (EVENT_FLAG_CTRLKEY), Shift (EVENT_FLAG_SHIFTKEY) or Alt (EVENT_FLAG_ALTKEY).
  • param - An optional parameter.

Once you set up the callback, OpenCV will invoke it whenever the mouse events are fired. To let the user select an object, you can write the on_mouse_move callback like so:

rectangle_points = []
def on_mouse_move(event, x, y, flags, param):
    # User pressed left mouse button and started drawing the rectangle    
    if(event == opencv.EVENT_LBUTTONDOWN):   
    # User has finished drawing the rectangle
    elif event == opencv.EVENT_LBUTTONUP:

The above function stores two coordinates of the rectangle (two opposite corners) selected by the user in the rectangle_points list. When the user presses the left mouse button, the callback invokes two statements:


The first one clears the coordinate list and then appends the new element by reading the x and y parameters passed to the callback by OpenCV. These coordinates represent the first corner of the rectangle. Then, the user holds the left mouse button down, and moves the cursor to the opposite rectangle's corner. Then, the user has to release the left mouse button, which in turn, invokes the following statements from the on_mouse_move callback:


The first statement will append the x, y coordinates of the mouse cursor to the rectangle_points list. The second statement will display the Lena image using the following method:

def display_lena_image(draw_rectangle):
    # File path
    file_path = common.LENA_FILE_PATH
    # Load image
    lena_img = opencv.imread(file_path)
    # Draw rectangle
        draw_rectangle_and_label(lena_img, 'Lena')
    # Show image
    opencv.imshow(common.WINDOW_NAME, lena_img)

The display_lena_image function shown above uses OpenCV's imread method to read the image from the file. Then, the function checks if the input parameter draw_rectangle is True. If so, the display_lena_image will draw the rectangle (see below) selected by the user before showing the final image to the user. To display the image, I use the imshow method and waitKey to wait for the user to press any key.

Drawing the Rectangle and Label

To draw the rectangle selected by the user, I implemented the draw_rectangle_and_label function:

def draw_rectangle_and_label(img, label):
    opencv.rectangle(img, rectangle_points[0], rectangle_points[1], 
        common.GREEN, common.LINE_THICKNESS)
    text_origin = (rectangle_points[0][0], rectangle_points[0][1] - common.TEXT_OFFSET)
    opencv.putText(img, label, text_origin, 
        common.FONT_FACE, common.FONT_SCALE, common.GREEN, 
        common.FONT_THICKNESS, common.FONT_LINE)

The above method uses two OpenCV functions: rectangle and putText.

The first one, as the name implies, draws the rectangle.

The second function, putText, is used here to write the label above the rectangle.

Putting Things Together

I combined the above functions into one Python script, (see companion code):

import cv2 as opencv
import common
rectangle_points = []
def draw_rectangle_and_label(img, label):
    # Definition of the draw_rectangle_and_label
def display_lena_image(draw_rectangle):
    # Definition of the display_lena_image
def on_mouse_move(event, x, y, flags, param):
    # Definition of the on_mouse_move
# Prepare window and set mouse callback
opencv.setMouseCallback(common.WINDOW_NAME, on_mouse_move)
# Display Lena image

To test the app, just run The code will create the window, set the mouse callback, and then display the Lena image without any annotations. You then select the rectangle, and the app will draw it on the image as shown in the introduction.

Wrapping Up

We learned how to use mouse callbacks in OpenCV to select various objects in images. This functionality is commonly used to prepare test and training datasets, and to indicate detected objects. In the next article, we will use drawing functions to depict detected objects.

This article is part of the series 'AI Social Distancing Detector View All


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
United States United States
Dawid Borycki is a software engineer and biomedical researcher with extensive experience in Microsoft technologies. He has completed a broad range of challenging projects involving the development of software for device prototypes (mostly medical equipment), embedded device interfacing, and desktop and mobile programming. Borycki is an author of two Microsoft Press books: “Programming for Mixed Reality (2018)” and “Programming for the Internet of Things (2017).”

Comments and Discussions

-- There are no messages in this forum --