Step 3. How to Add Object Detection

Having people bounding box floating through the frame is nice, but not realistic. We want to build real object detection.

We will start implementing the detection and tracking, assuming that the video contains only one object and create multi-object detection and tracking later.

Before we continue, it is important to understand the following terms:

• Real-world object: something visible in front of a camera that is unclassified until someone makes a decision to classify it. For example, a face in view of a camera is nothing special until it’s recognized and classified by someone as a face.
• Object: a particular subset of pixels extracted from a video frame, according to a developed detection algorithm. The process of computational recognition of something in the video stream does not deal with real-world object, but rather with its representation in the digital video stream, which is just a stream of numbers, i.e. of video frames constituted of pixels. When a plugin code detects real-world object, in fact it detects a digital "trace" of real-world object and cuts a particular subset of pixels from a video frame, according to a developed detection algorithm. This subset of pixels we call “an object” throughout this manual.
• Оbject metadata: information about the detected object. The Server’s expected representation of the detected object has to have a special structure in the form of the ObjectMetadata class. Metadata includes bounding box, object type, and any other attributes detected by the algorithm.
• Object track: a series of digital traces of a real-world object, i.e. the series of detected objects, which can form a sequence, known as the object's digital track. The plugin determines whether several traces, i.e. objects, represent the same real-world object.

Installing and linking OpenCV framework

First, we need to make sure the default conan profile is configured correctly.

Add Conan Center to the remotes:

conan remote add conancenter https://center.conan.io

Configure these settings in the default conan profile:

conan profile new default
conan profile update settings.arch=x86_64 default
conan profile update settings.compiler=gcc default
conan profile update settings.compiler.version=12 default
conan profile update settings.compiler.libcxx=libstdc++11 default
conan profile update settings.build_type=Release default
conan profile update settings.os=Linux default 

Next, we need to add OpenCV to dependencies of the plugin.

Open CMakeLists.txt and make the following changes.

Insert the following section into this if-clause
if(metadataSdkDir STREQUAL "") if-clause underneath the set(metadataSdkDir... statement:

# Download OpenCV automatically using Conan
if(NOT EXISTS "${CMAKE_BINARY_DIR}/conan.cmake")
    message(STATUS "Downloading conan.cmake from https://github.com/conan-io/cmake-conan")
    file(DOWNLOAD "https://raw.githubusercontent.com/conan-io/cmake-conan/master/conan.cmake" 
        "${CMAKE_BINARY_DIR}/conan.cmake")
endif()
include(${CMAKE_BINARY_DIR}/conan.cmake)

if(WIN32)
    set(CONAN_VS_RUNTIME_MT_SETTING "compiler.runtime=MT") 
endif()
set(OPENCV_OPTIONS
    opencv:parallel=False
    opencv:contrib=True
    opencv:contrib_freetype=False
    opencv:contrib_sfm=False
    opencv:with_jpeg=False
    opencv:with_png=False
    opencv:with_tiff=False
    opencv:with_jpeg2000=False
    opencv:with_openexr=False
    opencv:with_eigen=False
    opencv:with_webp=False
    opencv:with_quirc=False
    opencv:with_cuda=False
    opencv:with_cublas=False
    opencv:dnn=True
)

if(UNIX)
    set(OPENCV_OPTIONS
    ${OPENCV_OPTIONS}
    opencv:fPIC=True
    opencv:with_gtk=False
    opencv:with_cufft=False
    opencv:with_v4l=False
    )
endif()

conan_cmake_run(BUILD_TYPE "Release")
conan_cmake_configure(REQUIRES opencv/4.1.2
    GENERATORS cmake_find_package
    OPTIONS ${OPENCV_OPTIONS}
    SETTINGS ${CONAN_VS_RUNTIME_MT_SETTING} 
)

conan_cmake_install(PATH_OR_REFERENCE .
    OPTIONS ${OPENCV_OPTIONS}
    SETTINGS ${CONAN_VS_RUNTIME_MT_SETTING} 
    BUILD missing
)

find_package(OpenCV REQUIRED)

Let’s collect all the necessary OpenCV submodule libraries and link these statically:

set(CMAKE_EXE_LINKER_FLAGS " -static")
target_link_libraries(opencv_object_detection_analytics_plugin
nx_kit
nx_sdk
opencv::core opencv::flann opencv::imgproc opencv::imgcodecs opencv::dnn opencv::opencv_dnn opencv::ml
opencv::plot opencv::opencv_features2d opencv::opencv_calib3d opencv::datasets opencv::video opencv::tracking
   -static-libgcc
   -static-libstdc++
)

Build the project. It will take a few minutes to download OpenCV from the repository. This operation is executed only once, CMake is going to use the downloaded version from cache in the following builds.

rm -rf $BUILD_DIR/*
cmake -DmetadataSdkDir=~/develop/metadata_sdk -B $BUILD_DIR ./step3
cmake --build $BUILD_DIR --config Release -- -j

Installing and linking MobileNet SSD Caffe models

Now we are going to install the MobileNet SSD Caffe model pre-trained for object detection. 

In the CMakeLists.txt file we insert the following code:

if(NOT DEFINED ARTIFACTORY_URL)
        set(ARTIFACTORY_URL "https://resources.vmsproxy.com/nx_open_integrations")
endif()

# Download model files
set(model_file_names
    MobileNetSSD.caffemodel
    MobileNetSSD.prototxt
)
foreach(model_file_name IN LISTS model_file_names)
    set(model_file "${CMAKE_BINARY_DIR}/${model_file_name}")
    list(APPEND model_files "${model_file}")
    if(NOT EXISTS "${model_file}")
        message("Downloading ${model_file_name}")
        file(DOWNLOAD
                "${ARTIFACTORY_URL}/opencv/${model_file_name}"
                "${model_file}"
                SHOW_PROGRESS
                STATUS DOWNLOAD_STATUS
        )
        list(GET DOWNLOAD_STATUS 0 STATUS_CODE)
        if(NOT STATUS_CODE EQUAL 0)
            file(REMOVE "${model_file}")
            message(FATAL_ERROR "Error downloading ${ARTIFACTORY_URL}/opencv/${model_file_name}")
        endif()
    elseif()
        message("${model_file_name} already exits")
    endif()
endforeach()

To download those files, click Build > Run Cmake.

Note: We could use YOLO3 instead, but MobileNet SSD was chosen because it run fast on the CPU.

Organizing third-party library files

All the third party libraries a plugin depends on should be placed on the Server in the same directory as the plugin binary. Generally, a library is supplied with one or several header files and a binary file:

• .so for shared and .a for static libraries in Linux,
• .lib and .dll files in Windows.

Header files have to be added to the include directory list. Binary files have to be linked statically or dynamically with the resulting plugin binary. In our case this is done by the CMake code shown above in the Installing and linking OpenCV framework section.

While putting those files in the default plugin directory works, it is bad practice. Instead, a separate directory should be created with the plugin’s name, omitting the lib* prefix.

The Server will try to load linked libraries from that directory first. Let’s create the directory that will be used for our plugin and models:

sudo systemctl stop networkoptix-mediaserver
sudo mkdir $SERVER_DIR/bin/plugins/opencv_object_detection_analytics_plugin
sudo rm $SERVER_DIR/bin/plugins/libopencv_object_detection_analytics_plugin.so
sudo cp $BUILD_DIR/libopencv_object_detection_analytics_plugin.so $SERVER_DIR/bin/plugins/opencv_object_detection_analytics_plugin
sudo cp $BUILD_DIR/MobileNetSSD.caffemodel $BUILD_DIR/MobileNetSSD.prototxt $SERVER_DIR/bin/plugins/opencv_object_detection_analytics_plugin

As a result, we have the following directory structure:

$ ls -1 $SERVER_DIR/bin/plugins/opencv_object_detection_analytics_plugin
Libopencv_object_detection_analytics_plugin.so
MobileNetSSD.caffemodel
MobileNetSSD.prototxt

Obtaining plugin’s home directory

To load the model, we need to know its directory. While absolute paths can be used on the developer machine, they cannot be used elsewhere since the paths wouldn’t be valid on other computers.

The Metadata SDK provides the homeDir() method of nx::sdk::analytics::Plugin::utilityProvider to help detect the correct path.

Add the std::filesystem::path pluginHomeDir argument to the constructor of Engine and DeviceAgent. Additionally, place it in Engine as the m_pluginHomeDir private member. Apply the changes indicated in bold within each of the files below:

Updating engine.h

class Engine: public nx::sdk::analytics::Engine
{
public:
    explicit Engine(std::filesystem::path pluginHomeDir);
…

private:
    std::filesystem::path m_pluginHomeDir;
};

Updating engine.cpp 

Engine::Engine(std::filesystem::path pluginHomeDir):
// Call the DeviceAgent helper class constructor telling it to verbosely report to stderr.
    nx::sdk::analytics::Engine(/*enableOutput*/ true),
    m_pluginHomeDir(pluginHomeDir)
{
}
void Engine::doObtainDeviceAgent(Result<IDeviceAgent*>* outResult, const IDeviceInfo* deviceInfo)
{
    *outResult = new DeviceAgent(deviceInfo, m_pluginHomeDir);
}

Updating device_agent.h

public:
DeviceAgent(
     const nx::sdk::IDeviceInfo* deviceInfo,
     std::filesystem::path pluginHomeDir);

Updating device_agent.cpp

DeviceAgent::DeviceAgent(
    const nx::sdk::IDeviceInfo* deviceInfo,
    std::filesystem::path pluginHomeDir):
// Call the DeviceAgent helper class constructor telling it to verbosely report to stderr.
VideoFrameProcessingDeviceAgent(deviceInfo, /*enableOutput*/ true)
{
}

Updating plugin.cpp

Result<IEngine*> Plugin::doObtainEngine()
{
    const auto utilityProvider = this->utilityProvider();
    const std::filesystem::path pluginHomeDir = utilityProvider->homeDir();
    return new Engine(pluginHomeDir);
}

Structures for object detection

We are going to use the OpenCV library and models for object detection. They operate with data structures that are not compatible with those processed by the Server.

To pass frame information across different formats, we create the following header files and declare the helper structures using the code below: 

• detection.h - for storing VMS Server specific data
• frame.h - for storing OpenCV specific data

detection.h 

#include <nx/sdk/analytics/rect.h>
#include <nx/sdk/uuid.h>
struct Detection
{
    nx::sdk::analytics::Rect boundingBox;
    std::string classLabel;
    float confidence;
    nx::sdk::Uuid trackId;
};

frame.h

#include <opencv2/core/core.hpp>
#include <nx/sdk/analytics/i_uncompressed_video_frame.h>
struct Frame
{
    const int width;
    const int height;
    const int64_t timestampUs;
    const int64_t index;
    const cv::Mat cvMat;
public:
    Frame(const nx::sdk::analytics::IUncompressedVideoFrame* frame, int64_t index):
        width(frame->width()),
        height(frame->height()),
        timestampUs(frame->timestampUs()),
        index(index),
        cvMat({
            /*_rows*/ height,
            /*_cols*/ width,
            /*_type*/ CV_8UC3, //< BGR color space (default for OpenCV)
            /*_data*/ (void*) frame->data(0),
            /*_step*/ (size_t) frame->lineSize(0),
        })
    {
    }
};

IMPORTANT: For further reading past this point the full source code is needed. We are going to observe the principal plugin related part only, omitting OpenCV framework specifics. The code examples below are informational and are not supposed to be copy/pasted anywhere.

For full source code details, see files in opencv_object_detection_analytics_plugin/step3/src/sample_company/vms_server_plugins/opencv_object_detection.

To detect objects we need to initialize the model. The ObjectDetector class is responsible for loading the model, object detection and conversion from internal OpenCV representation to Detection structure, and termination in case of errors.

That is why the Detection structure was declared. It is used to store the objects inside our plugin and to convert them to ObjectMetadata just before invoking the pushMetadataPacket().

The Frame structure will be used for converting frame data from Server IUncompressedVideoFrame class to OpenCV cv::Mat class format.

Error handling

We need to think about possible errors that might occur and how they will be handled to make the plugin work reliably, so we introduce a hierarchy of exceptions to handle the errors. All exceptions in the code should be caught and converted to either 

• method result errors (for methods that support that) or
• plugin diagnostic event by invoking of pushPluginDiagnosticEvent().

Plugin diagnostic event is a VMS event that can be triggered by a Plugin to inform the System about some issue or status change.

We define all types of exception classes in exceptions.h:

class Error: public std::runtime_error { using std::runtime_error::runtime_error; };
class ObjectDetectorError: public Error { using Error::Error; };
class ObjectDetectorInitializationError: public ObjectDetectorError { using ObjectDetectorError::ObjectDetectorError; };
class ObjectDetectorIsTerminatedError: public ObjectDetectorError { using ObjectDetectorError::ObjectDetectorError; };
class ObjectDetectionError: public ObjectDetectorError { using ObjectDetectorError::ObjectDetectorError; };
inline std::string cvExceptionToStdString(const cv::Exception& e)
{
    return "OpenCV error: " + e.err + " (error code: " + std::to_string(e.code) + ")";
}

pushPluginDiagnosticEvent() is the method of the ConsumingDeviceAgent helper class. This method sends an event to the Server, so the user will see a notification about the plugin's state. 

Preventing error flooding

If the code raises an exception and doesn’t catch it, the Server will crash, but if the plugin catches all exceptions, sends plugin diagnostic events, but keeps executing the faulty code, it will flood the Server and client with notifications. So we must prevent code execution inside the DeviceAgent if it cannot recover from the error.

We introduce a boolean DeviceAgentclass member variable m_terminated to indicate that DeviceAgent should not execute any code. When m_terminated is set to true, DeviceAgent won’t execute.

We also store the previous value of DeviceAgent::m_terminated in m_terminatedPrevious to report an error only once, in order not to flood the Server with errors. We need to check the m_terminated variable in each method that could be called by the Server. 

As you may remember from the Plugin control flow diagram, the Server will send every frame to the plugin by calling pushDataPacket(), which in turn invokes pushUncompressedVideoFrame() of the ConsumingDeviceAgent helper class to supply the plugin with a frame for processing and object detection. Inside that function the plugin should perform detection and then send object metadata back to the Server.

Here how it looks inside the code:

bool DeviceAgent::pushUncompressedVideoFrame(const IUncompressedVideoFrame* videoFrame)
{
    m_terminated = m_terminated || m_objectDetector->isTerminated();
    if (m_terminated)
    {
        if (!m_terminatedPrevious)
        {
             pushPluginDiagnosticEvent(
                 IPluginDiagnosticEvent::Level::error,
                 "Plugin is in broken state.",
                 "Disable the plugin.");
             m_terminatedPrevious = true;
        }
        return true;
}
...
    const MetadataPacketList metadataPackets = processFrame(videoFrame);
...
}

• The m_terminated value is calculated every time pushUncompressedVideoFrame() is called.
• The status of ObjectDetector is checked by calling its isTerminated()method.
• The state of DeviceAgentis stored in the corresponding m_terminated field.
• The previous value of DeviceAgent::m_terminated is stored in m_terminatedPrevious to report about error only once.
• If DeviceAgent should be terminated, a corresponding message is sent to the Desktop client by calling pushPluginDiagnosticEvent() and the pushUncompressedVideoFrame() returns without doing anything.
• All exceptions related to object detection are triggered inside the ObjectDetector class and are caught in DeviceAgent->processFrame()

Object detection error handling

Besides general code execution errors, we have to handle object detection errors as well. We will handle them in the DeviceAgent::processFrame() method.

try
{
     DetectionList detections = m_objectDetector->run(frame);
     const auto& objectMetadataPacket =
         detectionsToObjectMetadataPacket(detections,frame.timestampUs);
     MetadataPacketList result;
     if (objectMetadataPacket)
         result.push_back(objectMetadataPacket);
     return result;
}
catch (const ObjectDetectionError& e)
{
     pushPluginDiagnosticEvent(
         IPluginDiagnosticEvent::Level::error,
         "Object detection error.",
         e.what());
     m_terminated = true;
}

• An exception is caught and converted to the resulting error sent to the Desktop client.
• The state of DeviceAgentis stored to the m_terminated field, thus preventing code from being executed next time pushUncompressedVideoFrame() is called.  

Skipping frames during processing

In real-life scenarios there might be a number of reasons why it is not needed to process every frame and it is reasonable to skip some of them. Here we'll learn how to do it.

We store the metadata packet with detection parameters in the class field m_detectionsMetadataPacket. It preserves the information about detections between DeviceAgent::pushUncompressedVideoFrame() calls each time we bind the metadata packet to the last video frame using the timestamp (in milliseconds). 

Since we access the metadata packet using SDK reference countable smart pointers (see IRefCountable in the SDK documentation), we need to call addRef() method to make the smart pointer know that the object is used somewhere. We release the data using releasePtr() just before the new detection. 

bool DeviceAgent::pushUncompressedVideoFrame(const IUncompressedVideoFrame* videoFrame)
{
...
    m_lastVideoFrameTimestampUs = videoFrame->timestampUs();
    // Detecting objects only on every `kDetectionFramePeriod` frame.
    if (m_frameIndex % kDetectionFramePeriod == 0)
    {
        const MetadataPacketList metadataPackets = processFrame(videoFrame);
        for (const Ptr<IMetadataPacket>& metadataPacket: metadataPackets)
        {
            metadataPacket->addRef();
            pushMetadataPacket(metadataPacket.get());
        }
    }
    ++m_frameIndex;
    return true;
}

Note: An optimal value of kDetectionFramePeriod is a balance between video processing FPS (frames per second) and computer performance. Higher FPS value improves the tracking accuracy of fast-moving objects, while lower FPS reduces resources requirements and enables the processing of more cameras on the same hardware.

Often 10 FPS is sufficient for object detection and tracking, and 1 FPS is good for object classification (extracting additional metadata information). When developing real-world video analytics applications, you need to consider these factors to choose the optimal FPS value.

Also, it is recommended to use adaptive value and drop more frames if the computer performance is insufficient.

ObjectDetector initialization

For detecting objects we declared and implemented the ObjectDetector class (see object_detector.cpp and object_detector.h). It has several public methods:

• explicit ObjectDetector(std::filesystem::path modelPath);
• void ensureInitialized();
• bool isTerminated() const;
• void terminate();
• DetectionList run(const Frame& frame);

The heart of the ObjectDetector class is the runImpl() private method. The method does actually utilize OpenCV framework functions for object detection. Before being used, the ObjectDetector class has to initialize the OpenCV framework and load object detection models.

We have to do it only once. It is what ensureInitialized() method is for. The right place for initialization (i.e. ensureInitialized() being called) is DeviceAgent::doSetNeededMetadataTypes().

This method is called when the Server updates the metadata types that our plugin should send. This method can be called more than once, so to prevent the framework from being initialized each time, ObjectDetector::ensureInitialized() is implemented in a way to remember that it’s been already initialized before.

void DeviceAgent::doSetNeededMetadataTypes(
    nx::sdk::Result<void>* outValue,
    const nx::sdk::analytics::IMetadataTypes* /*neededMetadataTypes*/)
{
    if (m_terminated)
         return;
    try
    {
         m_objectDetector->ensureInitialized();
    }
    catch (const ObjectDetectorInitializationError& e)
    {
         *outValue = {ErrorCode::otherError, new String(e.what())};
         m_terminated = true;
    }
    catch (const ObjectDetectorIsTerminatedError& /*e*/)
    {
         m_terminated = true;
    }
};

ObjectDetector::ensureInitialized() can throw several exceptions that’s why we need to catch them here. Those are defined in exceptions.h.

void ObjectDetector::ensureInitialized()
{
    if (isTerminated())
    {
       throw ObjectDetectorIsTerminatedError(
            "Object detector initialization error: object detector is terminated.");
    }
    if (m_netLoaded)
        return;
    try
    {
        loadModel();
    }
    catch (const cv::Exception& e)
    {
        terminate();
        throw ObjectDetectorInitializationError("Loading model: " + cvExceptionToStdString(e));
    }
    catch (const std::exception& e)
    {
        terminate();
        throw ObjectDetectorInitializationError("Loading model: Error: "s + e.what());
    }
}

In the loadModel() function, we initialize the models that were downloaded on step "Installing and linking MobileNet SSD Caffe models":

void ObjectDetector::loadModel()
{
// Prepare paths of model weights and definition.
static const auto modelBin = m_modelPath /
     std::filesystem::path("MobileNetSSD.caffemodel");
static const auto modelTxt = m_modelPath /
     std::filesystem::path("MobileNetSSD.prototxt");
// Load the model for future processing using OpenCV.
m_net = std::make_unique<Net>(
     readNetFromCaffe(modelTxt.string(), modelBin.string()));
// Save the whether the net is loaded or not to prevent unnecessary load.
m_netLoaded = !m_net->empty();
if (!m_netLoaded)
     throw ObjectDetectorInitializationError("Loading model: network is empty.");
}

In the former method we use m_modelPath which is received from the DeviceAgent constructor as pluginHomeDir parameter containing plugin home directory (see "Obtaining plugin’s home directory"):

DeviceAgent::DeviceAgent(
const nx::sdk::IDeviceInfo* deviceInfo,
std::filesystem::path pluginHomeDir):
// Call the DeviceAgent helper class constructor telling it to verbosely report to stderr.
VideoFrameProcessingDeviceAgent(deviceInfo, /*enableOutput*/ true),
m_objectDetector(std::make_unique<ObjectDetector>(pluginHomeDir))
{
}

Compare to the ObjectDetector constructor

ObjectDetector::ObjectDetector(std::filesystem::path modelPath):
m_modelPath(std::move(modelPath))
{
}

Configuring objects to be detected

In the ObjectDetector class we use the standard way of running the neural network inference and decoding the results. We will mention the key points. 

MobileNet SSD Cafe is capable of detecting objects of certain types identified by string values. They are built into the model. We enumerate all these values in kClasses. At the moment we want only three of them to be detected, so we also define kClassesToDetect. See the detection.cpp:

static std::string kClasses[] = {
"background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat",
"chair", "cow", "dining table", "dog", "horse", "motorbike", "person", "potted plant",
"sheep", "sofa", "train", "tv monitor"
};
const std::vector<std::string> kClassesToDetect{"cat", "dog", "person"};

While processing the detection we will filter only objects that have a class of interest in kClassesToDetect. Here is an example of what the code might look like: 

const std::string classLabel = kClasses[(size_t) classIndex];
bool oneOfRequiredClasses = std::find(
     kClassesToDetect.begin(), kClassesToDetect.end(), classLabel) != kClassesToDetect.end();
if (confidentDetection && oneOfRequiredClasses)
{ ...

There are several specific notions the attention should be paid to. 

• Confidence level: a value representing the likelihood that an object belongs to the class of interest. Usually, it technically is represented by a number, a percentage, or a decimal fraction value.
• Confidence threshold: a confidence value, which is sufficient and acceptable for the particular task being solved by the computing program. This value is chosen by a user. In our case, the confidence threshold determines whether the object can be considered reliably detected as belonging to the specific class. 

A low confidence threshold value will be less accurate but provide more detections. Keep in mind that setting it too low may increase the number of false positives (misidentified objects) due to being too lenient.

A high confidence threshold value will be more accurate but provide fewer detections. Keep in mind that setting it too high may increase the number of false negatives (unidentified objects that should have been identified) due to being too stringent.

The hard-coded value that we use for the confidence threshold is found in convertRawDetectionToDetection() of object_detection.cpp, because the plugin has no settings yet (we will introduce them in the following steps).

static constexpr float confidenceThreshold = 0.5F;
const int& i = detectionIndex;
const float confidence = rawDetections.at<float>(i, (int) OutputIndex::confidence);
const auto classIndex = (int) (rawDetections.at<float>(i, (int) OutputIndex::classIndex));
const std::string classLabel = kClasses[(size_t) classIndex];
const bool confidentDetection = confidence >= confidenceThreshold;

Now It’s very easy to detect other classes of objects that are included in kClasses, we just need to add them to kClassesToDetect.

Detecting objects

As we know the Server feeds frames to our plugin by calling DeviceAgent::pushUncompressedVideoFrame(). This is the place we should perform actual frame processing and object detection. For convenience, we define DeviceAgent::processFrame().

DeviceAgent::MetadataPacketList DeviceAgent::processFrame(
const IUncompressedVideoFrame* videoFrame)
{
const Frame frame(videoFrame, m_frameIndex);
try
{
     DetectionList detections = m_objectDetector->run(frame);
     const auto& objectMetadataPacket =
         detectionsToObjectMetadataPacket(detections, frame.timestampUs);
     MetadataPacketList result;
     if (objectMetadataPacket)
         result.push_back(objectMetadataPacket);
     return result;
}
catch (const ObjectDetectionError& e)
{
     pushPluginDiagnosticEvent(
         IPluginDiagnosticEvent::Level::error,
         "Object detection error.",
         e.what());
     m_terminated = true;
}
return {};
}

As you can see we create a Frame structure (see "Structures for object detection") and pass it to the ObjectDetector for processing

DetectionList detections = m_objectDetector->run(frame);

which returns a vector of Detection structures.

Afterwards, we convert DetectionList to objectMetadataPacket

const auto& objectMetadataPacket =
detectionsToObjectMetadataPacket(detections, frame.timestampUs);

and return it to the Server.

if (objectMetadataPacket)
       result.push_back(objectMetadataPacket);
return result;

Summary

Let’s sum up what we did:

• Installed OpenCV framework and object detection MobileNet SSD models.
• Created the plugin home directory for storing our plugin binary and models we need for object detection.
• Changed the code for the receiving path to the home directory and storing in the pluginHomeDir variable for further referencing during plugin work.
• Defined the Detection and Frame structures.
• Defined the ObjectDetector class.
• Added member m_objectDetector (pointer to ObjectDetector class) to DeviceAgent class.

This is how our plugin is supposed to work now:

• The DeviceAgent constructor creates an instance of ObjectDetector and passes pluginHomeDir: m_objectDetector(std::make_unique<ObjectDetector>(pluginHomeDir))
• A class instance referred by m_objectDetector initializes itself by ensureInitialized() method invoked in DeviceAgent::doSetNeededMetadataTypes()
• In ensureInitialized(), the ObjectDetector instance invokes its private method loadModel(), where object detection models are loaded.
• When the Server sends to our plugin a new uncompressed frame from a camera, it calls pushUncompressedVideoFrame() which invokes the processFrame() method.
• The processFrame() method actually processes a frame and detects objects by calling m_objectDetector -> run().

Let’s build our plugin and see how it works.