In this article, I will see how to directly access the Inference result through the Probe function. We will use Python. However, since DeepStream's element plug-ins are developed in C/C++, the interface is achieved through Python Binding. Passing data between Python and element plugins developed in C/C++ through Python Binding is a bit tricky. To use it properly, you need to learn how to use it by referring to NVidia's manual.

Prerequisites

Several blogs have explained how to implement DeepStream using Python in Xavier NX. Since DeepStream can be used in all Jetson series using JetPack, the contents described here can be used in Jetson Nano, TX2, and Xavier. This time, we will cover the necessary contents to apply the DeepStream installed and tested earlier to your application. Please be sure to read the preceding articles first.

Python Bindings

You can find a detailed explanation in the DeepStream Python API Reference. Inference results can also be found in the Probe function.

Among the articles below, many parts related to Python binding were taken from Python Sample Apps Source Details.

MetaData

DeepStream MetaData contains inference results and other information used in analytics. The MetaData is attached to the Gst Buffer received by each pipeline component. The metadata format is described in detail in the SDK MetaData documentation and API Guide. The SDK MetaData library is developed in C/C++. Python bindings provide access to the MetaData from Python applications. The pyds.so module is available as part of the DeepStream SDK installation under /lib directory.

Memory Management

Memory for MetaData is shared by the Python and C/C++ code paths. For example, a MetaData item may be added by a probe function written in Python and needs to be accessed by a downstream plugin written in C/C++. The deepstream-test4 app contains such usage. The Python garbage collector does not have visibility into memory references in C/C++, and therefore cannot safely manage the lifetime of such shared memory. Because of this complication, Python access to MetaData memory is typically achieved via references without claiming ownership.

String Access

Some MetaData structures contain string fields. These are accessable in the following manner:

Setting a string field results in the allocation of a string buffer in the underlying C++ code.

obj.type = "Type"

This will cause a memory buffer to be allocated, and the string "TYPE" will be copied into it. This memory is owned by the C code and will be freed later. To free the buffer in Python code, use:

pyds.free_buffer(obj.type)

Tips : NvOSD_TextParams.display_text string now gets freed automatically when a new string is assigned.

Directly reading a string field returns C address of the field in the form of an int, for example:

obj = pyds.NvDsVehicleObject.cast(data);
print(obj.type)

This will print an int representing the address of obj.type in C (which is a char*).
To retrieve the string value of this field, use pyds.get_string(), for example:

print(pyds.get_string(obj.type))

Casting

Some MetaData instances are stored in GList form. To access the data in a GList node, the data field needs to be cast to the appropriate structure. This casting is done via cast() member function for the target type:

NvDsBatchMeta.cast
NvDsFrameMeta.cast
NvDsObjectMeta.cast
NvDsUserMeta.cast
NvDsClassifierMeta.cast
NvDsDisplayMeta.cast
NvDsLabelInfo.cast
NvDsEventMsgMeta.cast
NvDsVehicleObject.cast
NvDsPersonObject.cast

In version v0.5, standalone cast functions were provided. Those are now deprecated and superseded by the cast() functions above:

glist_get_nvds_batch_meta
glist_get_nvds_frame_meta
glist_get_nvds_object_meta
glist_get_nvds_user_meta
glist_get_nvds_classifier_meta
glist_get_nvds_display_meta
glist_get_nvds_label_info
glist_get_nvds_event_msg_meta
glist_get_nvds_vehicle_object
glist_get_nvds_person_object

Example:

l_frame = batch_meta.frame_meta_list
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)

Callback Function Registration

Custom MetaData added to NvDsUserMeta require custom copy and release functions. The MetaData library relies on these custom functions to perform deep-copy of the custom structure, and free allocated resources. These functions are registered as callback function pointers in the NvDsUserMeta structure. Callback functions are registered using these functions:

pyds.set_user_copyfunc(NvDsUserMeta_instance, copy_function)
pyds.set_user_releasefunc(NvDsUserMeta_instance, free_func)

Tips : Callbacks need to be unregistered with the bindings library before the application exits. The bindings library currently keeps global references to the registered functions, and these cannot last beyond bindings library unload which happens at application exit. Use the following function to unregister all callbacks:
pyds.unset_callback_funcs()

See the deepstream-test4 sample application for an example of callback registration and unregistration.
Limitation: The bindings library currently only supports a single set of callback functions for each application. The last registered function will be used.

Optimizations and Utilities

Python interpretation is generally slower than running compiled C/C++ code. To provide better performance, some operations are implemented in C and exposed via the bindings interface. This is currently experimental and will expand over time.
The following optimized functions are available:

pyds.NvOSD_ColorParams.set(double red, double green, double blue, double alpha)

This is a simple function that performs the same operations as the following:

txt_params.text_bg_clr.red = red
txt_params.text_bg_clr.green = green
txt_params.text_bg_clr.blue = blue
txt_params.text_bg_clr.alpha = alpha

These are performed on each object in deepstream_test_4.py, causing the aggregate processing time to slow down the pipeline. Pushing this function into the C layer helps to increase performance.

generate_ts_rfc3339 (buffer, buffer_size)

This function populates the input buffer with a timestamp generated according to RFC3339:

%Y-%m-%dT%H:%M:%S.nnnZ\0

Image Data Access

Decoded images are accessible as NumPy arrays via the get_nvds_buf_surface function. This function is documented in the API Guide.
Please see the deepstream-imagedata-multistream sample application for an example of image data usage.

What is Probe?

A needle-shaped tool made of metal is used as a kind of electrode and connected to a sensor to measure the temperature, vibration, and electrical changes during chemical changes.

<probe>

In DeepStream, we connect a probe function to understand what is happening in the nvdsosd element and update the data if necessary. GStreamer developers seem to have made this because this principle is reminiscent of probes used in measuring equipment such as multimeters and oscilloscopes that monitor electrical and electronic devices.

The nvosd element plugin draws bounding boxes, text, and region of interest (RoI) polygons using inference results. Therefore, by examining the nvosd element, information on an object recognized in a frame of a video or camera stream can be found.

Tips : Therefore, the most important function is the probe callback function when you want to make your application using DeepStream.

Changing the output image

In the picture above, the text on the upper right, which displays the frame information and the number of recognized objects, was added by the Probe function. Like this, the Probe function can change the content displayed by the nvosd element on the screen. The contents that can be changed are as follows.

Text output
Box color and color inside the box of classified objects

Text Output

The code related to text output in the Probe function is as follows.

        # Acquiring a display meta object. The memory ownership remains in
        # the C code so downstream plugins can still access it. Otherwise
        # the garbage collector will claim it when this probe function exits.
        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)
        display_meta.num_labels = 1
        py_nvosd_text_params = display_meta.text_params[0]
        # Setting display text to be shown on screen
        # Note that the pyds module allocates a buffer for the string, and the
        # memory will not be claimed by the garbage collector.
        # Reading the display_text field here will return the C address of the
        # allocated string. Use pyds.get_string() to get the string content.
        py_nvosd_text_params.display_text = "Frame Number={} Number of Objects={} Vehicle_count={} Person_count={} FPS={}".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON], (1 / (now - start)))

        # Now set the offsets where the string should appear
        py_nvosd_text_params.x_offset = 10
        py_nvosd_text_params.y_offset = 12

        # Font , font-color and font-size
        py_nvosd_text_params.font_params.font_name = "Serif"
        py_nvosd_text_params.font_params.font_size = 10
        # set(red, green, blue, alpha); set to White
        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0) # (red, green, blue , alpha)

        # Text background color
        py_nvosd_text_params.set_bg_clr = 1
        # set(red, green, blue, alpha); set to Black
        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)
        # Using pyds.get_string() to get display_text as string
        if prt:
            print(pyds.get_string(py_nvosd_text_params.display_text))
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)

You can specify information such as font type, text color, text background color, alpha value, and text position.

The following figure shows some changes to the text properties like this.

        # Font , font-color and font-size
        py_nvosd_text_params.font_params.font_name = "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf"
        py_nvosd_text_params.font_params.font_size = 20
        # set(red, green, blue, alpha); set to White
        py_nvosd_text_params.font_params.font_color.set(0.2, 0.2, 1.0, 1) # (red, green, blue , alpha)

        # Text background color
        py_nvosd_text_params.set_bg_clr = 1
        # set(red, green, blue, alpha); set to Black
        py_nvosd_text_params.text_bg_clr.set(0.2, 0.2, 0.2, 0.3)

Box color and color inside the box of classified objects

Next, we will change the color of the box that displays the classified objects. In the github example, only the color of the box is specified, but the background color can also be specified. There is also an advantage of applying an alpha channel.

            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                #obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
    
            except StopIteration:
                break

            obj_meta.rect_params.has_bg_color = 1
            obj_meta.rect_params.bg_color.set(0.0, 0.0, 1.0, 0.2) #It seems that only the alpha channel is working. RGB value is reflected.       
            obj_counter[obj_meta.class_id] += 1
            obj_meta.rect_params.border_color.set(0.0, 1.0, 1.0, 1.0)    #It seems that the alpha channel is not working.  (red, green, blue , alpha)

I set the background color to blue and gave it an alpha value of 0.2. However, the background color was not applied and only appeared in yellow green. The alpha value was applied properly. I haven't tested the same code in C/C++ code, so it's not accurate, but it's probably a problem with the DeepStream 5.0 or Python version of DeepStream . Conversely, the box color was painted normally, but the alpha value was not applied. In the case of the box border, the alpha value has no meaning, so it may have been intentionally implemented like this. I couldn't find an explanation for this part in the document.

Return value of Probe function

Finally, let's look at the return value of the probe function. This value also has a significant effect on the screen output. There are 4 possible return values. You can find the definition and description of return values at https://lazka.github.io/pgi-docs/Gst-1.0/enums.html#Gst.PadProbeReturn.

Gst.PadProbeReturn.DROP : drop data in data probes. For push mode this means that the data item is not sent downstream. For pull mode, it means that the data item is not passed upstream. In both cases, no other probes are called for this item and Gst.FlowReturn.OK or True is returned to the caller. It doesn't even draw a frame.
Gst.PadProbeReturn.OK : normal probe return value. This leaves the probe in place, and defers decisions about dropping or passing data to other probes, if any. If there are no other probes, the default behaviour for the probe type applies (‘block’ for blocking probes, and ‘pass’ for non-blocking probes).
Gst.PadProbeReturn.REMOVE : remove this probe. No more painting on the frame.
Gst.PadProbeReturn.HANDLED : pass the data item in the block probe and block on the next item.Do not paint in this frame.

Get Inference Results from DeepStream

The basic configuration for processing the result of inference in the probe function is as follows. In the code below, you can process the Inference result in the part commented out in blue.

def osd_sink_pad_buffer_probe(pad, info, u_data):
    gst_buffer = info.get_buffer()
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
	
    while l_frame is not None:
        try:	
	    frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
	except StopIteration:
            break

        frame_number=frame_meta.frame_num
        l_obj=frame_meta.obj_meta_list

        while l_obj is not None:			
            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
                #obj_meta contains Primary, Secondary Detection Results
                #Do What you want to do Here !!!!
                            
            except StopIteration:
                break
            try: 
                l_obj=l_obj.next
            except StopIteration:
                break	

        try:
            l_frame=l_frame.next
        except StopIteration:
            break	
			
	return Gst.PadProbeReturn.OK

Primary Detector Inference Results

The following are information values that can be obtained even when only Promary Detector is used. The most important information is class_id, which informs the classified object information, and rect_params, which informs the location and drawing information of the object.

Object	Class Type	Description
obj_meta.class_id	int	Index of the object class infered by the primary detector/classifier. VEHICLE=0, BICYCLE=1, PERSON=2, ROADSIGN=3
obj_meta.confidence	float	Not implemented in DeepStream 5.0
obj_meta.detector_bbox_info	pyds.NvDsObjectMeta.NvDsComp_BboxInfo	Not implemented in DeepStream 5.0
obj_meta.obj_label	str	an array to store the string describing the class of the detected object
obj_meta.object_id	int	Unique ID for tracking the object. @ref UNTRACKED_OBJECT_ID indicates the object has not been tracked
obj_meta.rect_params	pyds.NvDsObjectMeta.NvOSD_RectParams	Structure containing the positional parameters of the object in the frame. Can also be used to overlay borders / semi-transparent boxes on top of objects. Refer @see pyds.NvOSD_RectParams
obj_meta.rect_params.has_bg_color	int	Decide whether to paint the box background color or not. If set to 0, obj_meta.rect_params.bg_color is ignored
obj_meta.rect_params.bg_color	pyds.NvOSD_ColorParams	Specifies the background color of the box. It contains 4 values (red, green, blue, alpha). Each value has a float value between 0 and 1. If the red value 255 corresponds to 1.0.
obj_meta.rect_params.border_color	pyds.NvOSD_ColorParams	Specifies the color of the box border. It contains 4 values (red, green, blue, alpha). Each value has a float value between 0 and 1. If the red value 255 corresponds to 1.0.
obj_meta.rect_params.border_width	int	Indicate the width of the box.
obj_meta.rect_params.border_width	int	Indicate the width of the box. Thickness of Line
obj_meta.rect_params.left	int	Holds the box's left coordinate in pixels
obj_meta.rect_params.top	int	Holds the box's top coordinate in pixels
obj_meta.rect_params.width	int	Holds the box's width coordinate in pixels
obj_meta.rect_params.height	int	Holds the box's height coordinate in pixels.
obj_meta.mask_params	pyds.NvDsObjectMeta.NvOSD_MaskParams	Not binded for Python. Don't use this object until the function is implemented

Then, let's print out the result of the Primary Detector described in the table above. Then, let's print out the result of the Primary Detector described in the table above. The code is modified from deepstream_test2.py. I intentionally omitted the main function. The main function has not been modified. If you uncomment the commented part of analyze_meta function , you can check the values of the remaining members of obj_meta (NvDsObjectMeta class). In the example code, inference results are displayed every 30 frames.

#!/usr/bin/env python3

################################################################################
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

import sys, time
sys.path.append('../')
import platform
import configparser

import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst
from common.is_aarch_64 import is_aarch64
from common.bus_call import bus_call

import pyds, ctypes
import cv2
import numpy as np

PGIE_CLASS_ID_VEHICLE = 0
PGIE_CLASS_ID_BICYCLE = 1
PGIE_CLASS_ID_PERSON = 2
PGIE_CLASS_ID_ROADSIGN = 3

VEHICLE_TYPES = ['Coupe', 'LargeVehicle', 'Sedan', 'suv', 'truck', 'van']
CAR_MAKE  = ['acura', 'audi', 'bmw', 'chevrolet', 'chrysler', 'dodge', 'ford', 'gmc', 'honda', 'hyundai', 'infiniti', 'jeep', 'kia', 'lexus', 'mazda', 'mercedes', 'nissan', 'subaru', 'toyota', 'volkswagen']
CAR_COLOR = ['Black', 'Blue', 'brown', 'gold', 'green', 'grey','maroon', 'orange', 'red', 'silver', 'white', 'yellow' ]
PGIE_CLASS  = ['VEHICLE', 'BICYCLE', 'PERSON', 'ROADSIGN']
SECONDARY_ELEMENT = ['','Primary Detector', 'CarColor', 'CarMake', 'VehicleTypes']

def analyze_meta( frame_number, obj_meta):
    print(' === obj_meta Primary Detection [%d]==='%(frame_number))
    print('object ={%s}'%(PGIE_CLASS[obj_meta. class_id]))
    print('object_id={}'.format(obj_meta.object_id))
    print('object height={}'.format(obj_meta.rect_params.height))
    print('object left={}'.format(obj_meta.rect_params.left))
    print('object top={}'.format(obj_meta.rect_params.top))
    print('object width={}'.format(obj_meta.rect_params.width))
    
    
    '''
    print('class_id={}'.format(obj_meta. class_id))
    print('confidence={}'.format(obj_meta.confidence))
    #print('mask_params={}'.format(obj_meta.mask_params))
    print('obj_label={}'.format(obj_meta.obj_label))
    print('object_id={}'.format(obj_meta.object_id))
    print('rect_params={}'.format(obj_meta.rect_params))
    print('        rect_params bg_color alpha={}'.format(obj_meta.rect_params.bg_color.alpha))
    print('        rect_params bg_color blue={}'.format(obj_meta.rect_params.bg_color.blue))
    print('        rect_params bg_color green={}'.format(obj_meta.rect_params.bg_color.green))
    print('        rect_params bg_color red={}'.format(obj_meta.rect_params.bg_color.red))
   
    
    print('        rect_params border_color alpha={}'.format(obj_meta.rect_params.border_color.alpha))
    print('        rect_params border_color blue={}'.format(obj_meta.rect_params.border_color.blue))
    print('        rect_params border_color green={}'.format(obj_meta.rect_params.border_color.green))
    print('        rect_params border_color red={}'.format(obj_meta.rect_params.border_color.red))
    print('    rect_params border_width={}'.format(obj_meta.rect_params.border_width))
    print('    rect_params  color_id={}'.format(obj_meta.rect_params.color_id))
    print('    rect_params has_bg_color={}'.format(obj_meta.rect_params.has_bg_color))
    print('    rect_params has_color_info={}'.format(obj_meta.rect_params.has_color_info))
    print('    rect_params height={}'.format(obj_meta.rect_params.height))
    print('    rect_params left={}'.format(obj_meta.rect_params.left))
    print('    rect_params top={}'.format(obj_meta.rect_params.top))
    print('    rect_params width={}'.format(obj_meta.rect_params.width))
    print('    rect_params reserved={}'.format(obj_meta.rect_params.reserved))
    print('tracker_confidence={}'.format(obj_meta.tracker_confidence))
    '''
    
    ''' Not binded properly. detector_bbox_info, tracker_bbox_info
    print('detector_bbox_info={}'.format(dir(obj_meta.detector_bbox_info))) #NVDsComp_BboxInfo
    print('detector_bbox_info org_bbox_coords.left={}'.format(obj_meta.detector_bbox_info.org_bbox_coords.left))
    print('tracker_bbox_info={}'.format(dir(obj_meta.tracker_bbox_info))) #NVDsComp_BboxInfo
    '''


start = time.time()
prt = True
def osd_sink_pad_buffer_probe(pad,info,u_data):
    global start, prt
    frame_number=0
    #Intiallizing object counter with 0.
    obj_counter = {
        PGIE_CLASS_ID_VEHICLE:0,
        PGIE_CLASS_ID_PERSON:0,
        PGIE_CLASS_ID_BICYCLE:0,
        PGIE_CLASS_ID_ROADSIGN:0
    }
    num_rects=0
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    # Retrieve batch metadata from the gst_buffer
    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        now = time.time()
        try:
            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
            # The casting is done by pyds.NvDsFrameMeta.cast()
            # The casting also keeps ownership of the underlying memory
            # in the C code, so the Python garbage collector will leave
            # it alone.
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            
        except StopIteration:
            break

        frame_number=frame_meta.frame_num
        num_rects = frame_meta.num_obj_meta

        l_obj=frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
                if (frame_number %30 == 0):
                    obj_user = obj_meta.obj_user_meta_list
                    analyze_meta(frame_number, obj_meta)
                            
            except StopIteration:
                break
            obj_counter[obj_meta.class_id] += 1
            try: 
                l_obj=l_obj.next
            except StopIteration:
                break

        # Acquiring a display meta object. The memory ownership remains in
        # the C code so downstream plugins can still access it. Otherwise
        # the garbage collector will claim it when this probe function exits.
        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)
        display_meta.num_labels = 1
        py_nvosd_text_params = display_meta.text_params[0]
        # Setting display text to be shown on screen
        # Note that the pyds module allocates a buffer for the string, and the
        # memory will not be claimed by the garbage collector.
        # Reading the display_text field here will return the C address of the
        # allocated string. Use pyds.get_string() to get the string content.
        py_nvosd_text_params.display_text = "Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}  FPS={}".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON], (1 / (now - start)))

        # Now set the offsets where the string should appear
        py_nvosd_text_params.x_offset = 10
        py_nvosd_text_params.y_offset = 12

        # Font , font-color and font-size
        py_nvosd_text_params.font_params.font_name = "Serif"
        py_nvosd_text_params.font_params.font_size = 10
        # set(red, green, blue, alpha); set to White
        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)

        # Text background color
        py_nvosd_text_params.set_bg_clr = 1
        # set(red, green, blue, alpha); set to Black
        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)
        
        # Using pyds.get_string() to get display_text as string
        if(prt):
            print(pyds.get_string(py_nvosd_text_params.display_text))
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
        try:
            l_frame=l_frame.next
        except StopIteration:
            break
        start = now 
        prt = False
    return Gst.PadProbeReturn.OK

<deepstream_test2_primary.py>

Now run the code.

(python) spypiggy@XavierNX:~/src/deepstream_python_apps/apps/deepstream-test2$ python3 deepstream_test2_primary.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 

......
......

 === obj_meta Primary Detection [0]===
object ={VEHICLE}
object_id=1
object height=45.0
object left=585.0
object top=466.875
object width=60.0
 === obj_meta Primary Detection [0]===
object ={VEHICLE}
object_id=2
object height=42.1875
object left=540.0
object top=472.5
object width=48.0
 === obj_meta Primary Detection [0]===
object ={VEHICLE}
object_id=3
object height=81.5625
object left=612.0
object top=486.5625
object width=102.0
 === obj_meta Primary Detection [0]===
object ={PERSON}
object_id=4
object height=627.1875
object left=3.0
object top=396.5625
object width=198.0
 === obj_meta Primary Detection [0]===
object ={PERSON}
object_id=5
object height=376.875
object left=294.0
object top=452.8125
object width=159.0
Frame Number=0 Number of Objects=5 Vehicle_count=3 Person_count=2  FPS=0.01487494765818283

......
......

In the above frame 0 output screen, it can be seen that the classification of 5 objects (3 cars and 2 people) and the location and size of individual objects are accurately output.

Secondary Detector Inference Results

Example deepstream_test2 goes through a second inference that classifies car color, car type, and manufacturer, using the inference result passed through the Primary Detector. These two processes are connected to each other and must be connected to each other because the result value passed through the primary detector is used. For an explanation of the example deepstream_test2, refer to Xavier NX-DeepStream 5.0 #3-Run Python Samples (test2).

This time, I added the code to get the result of Secondary Detector.

The core of the Secondary Detector result is the NvDsClassifierMeta class.
For one Primary Detection result, 3 Secondary Detectors (car color, car maker, car type) operate. Therefore, up to 3 results can be obtained. Therefore, iteration is repeated to bring the result values of the Secondary Detectors. The number of result values is between 0 and 3. For how to repeat iteration, check directly with the code below.
And when iteration determines that there is a result value, the object of the NvDsLabelInfo class type is obtained using the pyds.glist_get_nvds_label_info function. This object contains the result of the Secondary Detector.

def analyze_meta( frame_number, obj_meta):
    print(' === obj_meta Primary Detection [%d]==='%(frame_number))
    print('object ={%s}'%(PGIE_CLASS[obj_meta. class_id]))
    print('object_id={}'.format(obj_meta.object_id))
    print('object height={}'.format(obj_meta.rect_params.height))
    print('object left={}'.format(obj_meta.rect_params.left))
    print('object top={}'.format(obj_meta.rect_params.top))
    print('object width={}'.format(obj_meta.rect_params.width))

    if obj_meta.class_id == PGIE_CLASS_ID_VEHICLE:     #Only vehicle supports secondary inference
        print(' === obj_meta Secondary Detection ===')
        cls_meta = obj_meta.classifier_meta_list
        while cls_meta is not None:
            cls = pyds.NvDsClassifierMeta.cast(cls_meta.data)
            
            #unique_component_id : Primary Detector:1, Secondary_CarColor: 2, Secondary_CarMake : 3,  Secondary_VehicleTypes:4
            #print('Secondary element={}'.format(SECONDARY_ELEMENT[cls.unique_component_id])) 
            
            #print(' num_labels={}'.format(cls.num_labels))
            info = cls.label_info_list  # type of pyds.GList
            while info is not None:
                label_meta = pyds.glist_get_nvds_label_info(info.data)
                if cls.unique_component_id == 2:
                    print('\tCarColor ={}'.format(CAR_COLOR[label_meta.result_class_id]))
                elif cls.unique_component_id == 3:    
                    print('\tCarMake ={}'.format(CAR_MAKE[label_meta.result_class_id]))
                elif cls.unique_component_id == 4:    
                    print('\tVehicleTypes ={}'.format(VEHICLE_TYPES[label_meta.result_class_id]))
                #print('label_meta label_id ={}'.format(label_meta.label_id))
                print('\tlabel_meta result_prob ={}'.format(label_meta.result_prob))

                try:
                    info=info.next
                except StopIteration:
                    break
        
            try:
                cls_meta=cls_meta.next
            except StopIteration:
                break

<deepstream_test2_secondary.py>

Run the code.

(python) spypiggy@XavierNX:~/src/deepstream_python_apps/apps/deepstream-test2$ python3 deepstream_test2_secondary.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 

......
......

 === obj_meta Primary Detection [1440]===
object ={VEHICLE}
object_id=1150
object height=22.5
object left=1176.0
object top=480.9375
object width=57.0
 === obj_meta Secondary Detection ===
 === obj_meta Primary Detection [1440]===
object ={VEHICLE}
object_id=1143
object height=33.75
object left=588.0
object top=483.75
object width=42.0
 === obj_meta Secondary Detection ===
 === obj_meta Primary Detection [1440]===
object ={VEHICLE}
object_id=1088
object height=115.3125
object left=636.0
object top=472.5
object width=147.0
 === obj_meta Secondary Detection ===
	CarColor =white
	label_meta result_prob =0.5112398266792297
	CarMake =nissan
	label_meta result_prob =0.9992345571517944
	VehicleTypes =suv
	label_meta result_prob =0.5845340490341187
 === obj_meta Primary Detection [1440]===
object ={VEHICLE}
object_id=1140
object height=126.5625
object left=834.0
object top=461.25
object width=210.0
 === obj_meta Secondary Detection ===
	CarColor =red
	label_meta result_prob =0.7972736358642578
	VehicleTypes =suv
	label_meta result_prob =0.7728081941604614
 === obj_meta Primary Detection [1440]===
object ={VEHICLE}
object_id=1152
object height=45.0
object left=1410.0
object top=486.5625
object width=126.0
 === obj_meta Secondary Detection ===
 === obj_meta Primary Detection [1440]===
object ={VEHICLE}
object_id=1153
object height=67.5
object left=1578.0
object top=483.75
object width=174.0
 === obj_meta Secondary Detection ===
	CarColor =white
	label_meta result_prob =0.9581888318061829
	VehicleTypes =truck
	label_meta result_prob =0.6332785487174988
 === obj_meta Primary Detection [1440]===
object ={PERSON}
object_id=1142
object height=67.5
object left=441.0
object top=478.125
object width=21.0
 === obj_meta Primary Detection [1440]===
object ={PERSON}
object_id=1032
object height=67.5
object left=411.0
object top=478.125
object width=24.0

From the recognition result of object_id value of 1088, it can be seen that the car maker, color, and type are all identified. However, in the case of object_id 1152, the Secondary Detector did not recognize anything.

Get frames available in OpenCV from DeepStream

This time, I will see how to check the result value recognized by the probe function as an image. Images delivered through GStreamer's pipeline can be checked directly through the probe function. I will explain how to integrate OpenCV, which we are familiar with.

The function to get the image from the probe function is done in the pyds.get_nvds_buf_surface function.

frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
img = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
img = cv2.cvtColor(img, cv2.COLOR_RGBA2BGR)

Tip : DeepStream uses RGBA channels for alpha channel processing. Therefore, change it to the BGR format used for OpenCV.

This time, I will make an example of saving the objects recognized earlier as files. A specific object is cut out from the original image by using the object size obtained from Primary Detection. And, in the case of a car, the result value obtained from Secondary Detection is added to the file name.

def analyze_meta(img, frame_number, obj_meta, save):
    filename = '/home/spypiggy/src/test_images/result/%d_%d_'%(frame_number, obj_meta.object_id)
    print(' === obj_meta Primary Detection [%d]==='%(frame_number))
    print('object ={%s}'%(PGIE_CLASS[obj_meta. class_id]))
    print('object_id={}'.format(obj_meta.object_id))
    print('object height={}'.format(obj_meta.rect_params.height))
    print('object left={}'.format(obj_meta.rect_params.left))
    print('object top={}'.format(obj_meta.rect_params.top))
    print('object width={}'.format(obj_meta.rect_params.width))
    filename += (PGIE_CLASS[obj_meta. class_id])
    
    if obj_meta.class_id == PGIE_CLASS_ID_VEHICLE:     #Only vehicle supports secondary inference
        print(' === obj_meta Secondary Detection ===')
        cls_meta = obj_meta.classifier_meta_list
        filename_ex = filename
        while cls_meta is not None:
            cls = pyds.NvDsClassifierMeta.cast(cls_meta.data)
            
            #unique_component_id : Primary Detector:1, Secondary_CarColor: 2, Secondary_CarMake : 3,  Secondary_VehicleTypes:4
            #print('Secondary element={}'.format(SECONDARY_ELEMENT[cls.unique_component_id])) 
            
            #print(' num_labels={}'.format(cls.num_labels))
            info = cls.label_info_list  # type of pyds.GList
            while info is not None:
                label_meta = pyds.glist_get_nvds_label_info(info.data)
                if cls.unique_component_id == 2:
                    print('\tCarColor ={}'.format(CAR_COLOR[label_meta.result_class_id]))
                    filename_ex += ('_'+CAR_COLOR[label_meta.result_class_id])
                elif cls.unique_component_id == 3:    
                    print('\tCarMake ={}'.format(CAR_MAKE[label_meta.result_class_id]))
                    filename_ex += ('_'+CAR_MAKE[label_meta.result_class_id])
                elif cls.unique_component_id == 4:    
                    print('\tVehicleTypes ={}'.format(VEHICLE_TYPES[label_meta.result_class_id]))
                    filename_ex += ('_'+VEHICLE_TYPES[label_meta.result_class_id])
                #print('label_meta label_id ={}'.format(label_meta.label_id))
                print('\tlabel_meta result_prob ={}'.format(label_meta.result_prob))
                filename_ex += ('_%4.2f'%(label_meta.result_prob))

                try:
                    info=info.next
                except StopIteration:
                    break
        
            try:
                cls_meta=cls_meta.next
            except StopIteration:
                break                        

        filename_ex += '.jpg'
        if save:
            crop_img = img[int(obj_meta.rect_params.top):int(obj_meta.rect_params.top)+int(obj_meta.rect_params.height), int(obj_meta.rect_params.left):int(obj_meta.rect_params.left)+int(obj_meta.rect_params.width)]
            cv2.imwrite(filename_ex, crop_img)
    else:  # save  'BICYCLE', 'PERSON', 'ROADSIGN'         
        filename_ex = filename + '.jpg'
        if save:
            crop_img = img[int(obj_meta.rect_params.top):int(obj_meta.rect_params.top)+int(obj_meta.rect_params.height), int(obj_meta.rect_params.left):int(obj_meta.rect_params.left)+int(obj_meta.rect_params.width)]
            cv2.imwrite(filename_ex, crop_img)
    


start = time.time()
prt = True
def osd_sink_pad_buffer_probe(pad,info,u_data):
    global start, prt
    frame_number=0
    #Intiallizing object counter with 0.
    obj_counter = {
        PGIE_CLASS_ID_VEHICLE:0,
        PGIE_CLASS_ID_PERSON:0,
        PGIE_CLASS_ID_BICYCLE:0,
        PGIE_CLASS_ID_ROADSIGN:0
    }
    num_rects=0
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    # Retrieve batch metadata from the gst_buffer
    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        now = time.time()
        try:
            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
            # The casting is done by pyds.NvDsFrameMeta.cast()
            # The casting also keeps ownership of the underlying memory
            # in the C code, so the Python garbage collector will leave
            # it alone.
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            img = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
            img = cv2.cvtColor(img, cv2.COLOR_RGBA2BGR)
        except StopIteration:
            break

        frame_number=frame_meta.frame_num
        num_rects = frame_meta.num_obj_meta
        if (frame_number %30 == 0): # Save Full image
            cv2.imwrite('/home/spypiggy/src/test_images/result/%d_original.jpg'%frame_number, img)
        
        l_obj=frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
                if (frame_number %30 == 0):
                    obj_user = obj_meta.obj_user_meta_list
                    analyze_meta(img, frame_number, obj_meta, True)
                            
            except StopIteration:
                break
            obj_counter[obj_meta.class_id] += 1
            try: 
                l_obj=l_obj.next
            except StopIteration:
                break

        # Acquiring a display meta object. The memory ownership remains in
        # the C code so downstream plugins can still access it. Otherwise
        # the garbage collector will claim it when this probe function exits.
        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)
        display_meta.num_labels = 1
        py_nvosd_text_params = display_meta.text_params[0]
        # Setting display text to be shown on screen
        # Note that the pyds module allocates a buffer for the string, and the
        # memory will not be claimed by the garbage collector.
        # Reading the display_text field here will return the C address of the
        # allocated string. Use pyds.get_string() to get the string content.
        py_nvosd_text_params.display_text = "Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}  FPS={}".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON], (1 / (now - start)))

        # Now set the offsets where the string should appear
        py_nvosd_text_params.x_offset = 10
        py_nvosd_text_params.y_offset = 12

        # Font , font-color and font-size
        py_nvosd_text_params.font_params.font_name = "Serif"
        py_nvosd_text_params.font_params.font_size = 10
        # set(red, green, blue, alpha); set to White
        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)

        # Text background color
        py_nvosd_text_params.set_bg_clr = 1
        # set(red, green, blue, alpha); set to Black
        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)
        
        # Using pyds.get_string() to get display_text as string
        if(prt):
            print(pyds.get_string(py_nvosd_text_params.display_text))
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
        try:
            l_frame=l_frame.next
        except StopIteration:
            break
        start = now 
        prt = False
    return Gst.PadProbeReturn.OK

Now when you run the program, you will see a lot of image files in the output directory.

Let's check the resulting images of frame 240.

<240_original.jpg>

The following image is a combination of image files storing objects recognized in frame 240 into one image. There are some errors in recognition, but we can see that we have handled the desired results correctly.

Wrapping Up

Due to the lack of examples using Python in DeepStream, it took a few days to create code to find the result of Secondary Detector and save it as an image. In the future, many Python examples will be introduced in DeepStream, so users who develop with Python are expected to be able to easily develop DeepStream-based solutions.

You can download the source code at https://github.com/raspberry-pi-maker/NVIDIA-Jetson .

NVIDIA Jetson and Raspberry Pi

2020년 8월 16일 일요일

Xavier NX-DeepStream 5.0 #4 - Handling Primary, Secondary Inference Results