Real-time Barcode Scanning from Camera Stream

Barcode detection and recognition technology, especially the application of real-time barcode scanning, is imperative to industries such as warehouse management and mobile payment. Dynamsoft Barcode Reader, as a cross-platform and cross-programing language SDK, is a lifesaver for developers who are dedicated to building enterprise-class barcode solutions in a short time.

In this post, we discuss how to leverage Dynamsoft Barcode Reader video decoding APIs to implement the barcode scanning functionality in camera preview scenario. Also, we demonstrate how to implement for desktop and mobile platforms respectively with the code snippet.

The Challenge of Camera Real-time Barcode Scanning

The process of reading barcodes consists of various computer vision algorithms, some of which are CPU-intensive. To make the camera preview run smoothly, you have to execute intensive computation asynchronously in a worker thread. A multithreading model is required for this case, in which developers have to spend a lot more time on thread scheduling, shared memory, status synchronization, and lock. Developers are prone to make mistakes in such a complicated scenario.

To eliminate the complexity of multi-threaded programming and improve developers’ efficiency, Dynamsoft Barcode Reader 7.0 brings a set of thread-based APIs for decoding continuous frames.

Dynamsoft Barcode Reader Overview

Dynamsoft's Barcode Reader SDK enables you to efficiently embed barcode reading functionality in your web, desktop or mobile application using just a few lines of code, which saves your development time and cost. With the SDK, you can create high-speed and reliable barcode scanner software to meet your business needs.

License

Get a FREE 30-day trial license.

Supported Barcode Symbologies

Linear Barcodes (1D): Code 39, Code 93, Code 128, Codabar, Interleaved 2 of 5, EAN-8, EAN-13, UPC-A, UPC-E, Industrial 2 of 5.

2D Barcodes: QR Code, DataMatrix, PDF417 and Aztec Code.

Developer’s Guide

https://www.dynamsoft.com/help/Barcode-Reader/devguide/index.html

API Documentation

https://www.dynamsoft.com/help/Barcode-Reader/index.html

Code Gallery

https://www.dynamsoft.com/Downloads/Dynamic-Barcode-Reader-Sample-Download.aspx

Online Demo

https://demo.dynamsoft.com/DBR/BarcodeReaderDemo.aspx

How Barcode Video Decoding APIs Work

The video decoding model contains a user main thread, a decoding thread, and a result consumer thread.

User Main Thread

The user main thread is initiated when StartFrameDecoding() is being called. Invoke AppendFrame() to add frames to an auto-managed frame queue in a video frame callback function.

Decoding Thread

The decoding thread takes all CPU-intensive work, running barcode algorithms. It uses some strategies to select and discard frames in order to avoid lagging.

Result Consumer Thread

This thread manages registered callback functions for fetching results.

Desktop Barcode Scanning

To build a desktop barcode app quickly, we recommend using Python and OpenCV.

Breaking Through Python’s GIL

The performance of Python multithreading code, especially for computation-intensive tasks, is restricted due to Python’s GIL (Global Interpreter Lock). The new video decoding APIs of Dynamsoft Barcode Reader is based on native C/C++ thread, which will not be affected by Python’s virtual machine.

Let’s take a glimpse of the Python sample code:

import cv2
import dbr
import time
import os
 
# The callback function for receiving barcode results
def onBarcodeResult(format, text):
    print("Type: " + format)
    print("Value: " + text + "\n")
 
def read_barcode():
    video_width = 640
    video_height = 480
 
    vc = cv2.VideoCapture(0)
    vc.set(3, video_width) #set width
    vc.set(4, video_height) #set height
 
    if vc.isOpened():  
        dbr.initLicense('LICENSE-KEY')
        rval, frame = vc.read()
    else:
        return
 
    windowName = "Barcode Reader"
 
    max_buffer = 2
    max_results = 10
    barcodeTypes = 0x3FF | 0x2000000 | 0x4000000 | 0x8000000 | 0x10000000 # 1D, PDF417, QRCODE, DataMatrix, Aztec Code
    image_format = 1 # 0: gray; 1: rgb888
 
    dbr.startVideoMode(max_buffer, max_results, video_width, video_height, image_format, barcodeTypes, onBarcodeResult)
 
    while True:
        cv2.imshow(windowName, frame)
        rval, frame = vc.read()
 
        start = time.time()
        try:
            ret = dbr.appendVideoFrame(frame)
        except:
            pass
 
        cost = (time.time() - start) * 1000
        print('time cost: ' + str(cost) + ' ms')
 
        # 'ESC' for quit
        key = cv2.waitKey(1)
        if key == 27:
            break
 
    dbr.stopVideoMode()
    dbr.destroy()
    cv2.destroyWindow(windowName)
 
if __name__ == "__main__":
    print("OpenCV version: " + cv2.__version__)
    read_barcode()

The dbr module is the Python extension built with Dynamsoft Barcode Reader. We created three methods: startVideoMode(), appendVideoFrame() and stopVideoMode(). The corresponding C/C++ code is as follows.

static PyObject *
startVideoMode(PyObject *self, PyObject *args)
{
    printf("Start the video mode\n");
    CHECK_DBR();
 
    PyObject *callback = NULL;
    int maxListLength, maxResultListLength, width, height, imageformat, iFormat, stride; 
    if (!PyArg_ParseTuple(args, "iiiiiiO", &maxListLength, &maxResultListLength, &width, &height, &imageformat, &iFormat, &callback)) {
        return NULL;
    }
 
    updateFormat(iFormat);
 
    if (!PyCallable_Check(callback)) 
    {
        PyErr_SetString(PyExc_TypeError, "parameter must be callable");
        return NULL;
    }
    else
    {
        Py_XINCREF(callback);         /* Add a reference to new callback */
        Py_XDECREF(py_callback);      /* Dispose of previous callback */
        py_callback = callback;     
    }
 
    ImagePixelFormat format = IPF_RGB_888;
 
    if (imageformat == 0)
    {
        stride = width;
        format = IPF_GRAYSCALED;
    }
    else {
        stride = width * 3;
        format = IPF_RGB_888;
    }
 
    DBR_SetTextResultCallback(hBarcode, onResultCallback, NULL);
 
    int ret = DBR_StartFrameDecoding(hBarcode, maxListLength, maxResultListLength, width, height, stride, format, "");
    return Py_BuildValue("i", ret);
}

appendVideoFrame()

static PyObject *
appendVideoFrame(PyObject *self, PyObject *args)
{
    CHECK_DBR();
 
    PyObject *o;
    if (!PyArg_ParseTuple(args, "O", &o))
        return NULL;
     
    #if defined(IS_PY3K)
    //Refer to numpy/core/src/multiarray/ctors.c
    Py_buffer *view;
    int nd;
    PyObject *memoryview = PyMemoryView_FromObject(o);
    if (memoryview == NULL) {
        PyErr_Clear();
        return -1;
    }
 
    view = PyMemoryView_GET_BUFFER(memoryview);
    unsigned char *buffer = (unsigned char *)view->buf;
    nd = view->ndim;
    int len = view->len;
    int stride = view->strides[0];
    int width = view->strides[0] / view->strides[1];
    int height = len / stride;
    #else
 
    PyObject *ao = PyObject_GetAttrString(o, "__array_struct__");
 
    if ((ao == NULL) || !PyCObject_Check(ao)) {
        PyErr_SetString(PyExc_TypeError, "object does not have array interface");
        return NULL;
    }
 
    PyArrayInterface *pai = (PyArrayInterface*)PyCObject_AsVoidPtr(ao);
     
    if (pai->two != 2) {
        PyErr_SetString(PyExc_TypeError, "object does not have array interface");
        Py_DECREF(ao);
        return NULL;
    }
 
    // Get image information
    unsigned char *buffer = (unsigned char *)pai->data; // The address of image data
    int width = (int)pai->shape[1];       // image width
    int height = (int)pai->shape[0];      // image height
    int stride = (int)pai->strides[0]; // image stride
    #endif
 
    // Initialize Dynamsoft Barcode Reader
    TextResultArray *paryResult = NULL;
 
    // Detect barcodes
    ImagePixelFormat format = IPF_RGB_888;
 
    if (width == stride) 
    {
        format = IPF_GRAYSCALED;
    }
    else if (width == stride * 3) 
    {
        format = IPF_RGB_888;
    }
    else if (width == stride * 4)
    {
        format = IPF_ARGB_8888;
    }
 
    int frameId = DBR_AppendFrame(hBarcode, buffer);
    return 0;
}

stopVideoMode()

static PyObject *
stopVideoMode(PyObject *self, PyObject *args)
{
    printf("Stop the video mode\n");
    if (hBarcode) 
    {
        int ret = DBR_StopFrameDecoding(hBarcode);
        return Py_BuildValue("i", ret);
    }
 
    return 0;
}

You have to call Python code between PyGILState_Ensure() and PyGILState_Release() when executing the callback function on the native thread:

void onResultCallback(int frameId, TextResultArray *pResults, void * pUser)
{
    // Get barcode results
    int count = pResults->resultsCount;
    int i = 0;
 
    // https://docs.python.org/2/c-api/init.html
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();
 
    for (; i < count; i++)
    {
        // https://docs.python.org/2.5/ext/callingPython.html
        PyObject *result = PyObject_CallFunction(py_callback, "ss", pResults->results[i]->barcodeFormatString, pResults->results[i]->barcodeText);
        if (result == NULL) return NULL;
        Py_DECREF(result);
    }
 
    PyGILState_Release(gstate);
    /////////////////////////////////////////////
 
    // Release memory
    DBR_FreeTextResults(&pResults);
}

Source Code

https://github.com/dynamsoft-dbr/python/tree/7.x

Android Barcode Scanning

To build a simple Android barcode scanning app with Android camera API and Dynamsoft Barcode Reader.

cameraView.addFrameProcessor(new FrameProcessor() {
            @SuppressLint("NewApi")
            @Override
            public void process(@NonNull final Frame frame) {
                try {
                    if (isDetected && isCameraOpen) {
                        YuvImage yuvImage = new YuvImage(frame.getData(), ImageFormat.NV21,
                                frame.getSize().getWidth(), frame.getSize().getHeight(), null);
                        if (width == 0) {
                            width = yuvImage.getWidth();
                            height = yuvImage.getHeight();
                            stride = yuvImage.getStrides()[0];
                            reader.setErrorCallback(errorCallback, null);
                            reader.setTextResultCallback(textResultCallback, null);
                            reader.setIntermediateResultCallback(intermediateResultCallback, null);
                            reader.startFrameDecoding(10, 10, width, height, stride, EnumImagePixelFormat.IPF_NV21, "");
                        } else {
                            PublicRuntimeSettings s  = reader.getRuntimeSettings();
                            int frameid = reader.appendFrame(yuvImage.getYuvData());
                            Log.i("FrameId", frameid + "");
                        }
                        if (bUpateDrawBox) {
                            bUpateDrawBox = false;
                            Message message = handler.obtainMessage();
                            Rect imageRect = new Rect(0, 0, width, height);
                            message.obj = imageRect;
                            message.what = 0x001;
                            handler.sendMessage(message);
                        }
                        isDetected = true;
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });

Call startDecodingFrame() to initialize the environment and call appendFrame() repeatedly to add the preview images.

Source Code

https://github.com/dynamsoft-dbr/android-barcode-decode-video

iOS Barcode Scanning

To build a simple iOS barcode scanning app with iOS camera API and Dynamsoft Barcode Reader.

(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection;
{
   isCurrentFrameDecodeFinished = NO;
   CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
   CVPixelBufferLockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);
   int bufferSize = (int)CVPixelBufferGetDataSize(imageBuffer);
   void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
   CVPixelBufferUnlockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);
   NSData * buffer = [NSData dataWithBytes:baseAddress length:bufferSize];
   startRecognitionDate = [NSDate date];
   if (width != (int)CVPixelBufferGetWidth(imageBuffer)) {
       width = (int)CVPixelBufferGetWidth(imageBuffer);
       height = (int)CVPixelBufferGetHeight(imageBuffer);
       stride = CVPixelBufferGetBytesPerRow(imageBuffer);
       [m_barcodeReader setDBRErrorDelegate:self userData:nil];
       [m_barcodeReader setDBRTextResultDelegate:self userData:nil];
       [m_barcodeReader setDBRIntermediateResultDelegate:self userData:nil];
       [m_barcodeReader startFrameDecoding:10 maxResultQueueLength:100 width:width height:height stride:stride format:EnumImagePixelFormatARGB_8888 templateName:@"" error:nil];
   } else {
       [m_barcodeReader appendFrame:buffer];
   }
}

Call the following to initialize the environment and call -(NSInteger)appendFrame:(NSData*) bufferBytes repeatedly to add the preview images.

(void)startFrameDecoding:(NSInteger)maxQueueLength
                     maxResultQueueLength:(NSInteger)maxResultQueueLength
                     width:(NSInteger)width
                     height:(NSInteger)height
                     stride:(NSInteger)stride
                     format:(EnumImagePixelFormat)format
                     templateName:(NSString* _Nonnull)templateName
                     error:(NSError* _Nullable * _Nullable)error

Source Code

https://github.com/dynamsoft-dbr/ios-barcode-decode-video

Technical Support

If you have any questions about Dynamsoft Barcode Reader SDK, please feel free to contact support@dynamsoft.com.

Release History

v7.0, 07/12/2019

New

• Refactored most modules to provide a flexible barcode reading framework that allows for parameter customization suited for a variety of barcode scenarios
• Enabled access to intermediate results (grayscale image, binarized image, text zone, etc) during the decoding process
• Added new interfaces to support video decoding, and frame decoding to improve interactive sensitivity
• Provided methods to terminate the decoding process at different phases such as during binarization, localization or barcode type identification.
• Added a new barcode localization method, Scan Directly, to reduce decoding time significantly for high-quality images.

Improved

• Enhanced error messages related to license initiation failure.
• Improved detailed results for decoded barcodes, including more barcode format specification
• Improved results output to enable outputting barcode results in the order of confidence level, barcode position or format.

Fix

• Fixed an issue where the barcode could be calculated incorrectly on some occasions.