This guide explains how to implement object tracking using the OpenCV Tracking by Matching (tbm) module.

Overview

Currently, the integration processes only the first confident detection per frame and skips all others. This creates two problems:

• Multiple objects can appear in a single frame.

• Objects move across frames, generating new detections.

To resolve this, the integration must track coordinate changes across frames. By assigning a unique identifier (trackId) to a series of matching detections, the server treats them as a single object. This ensures the object appears only once in search results, and the Desktop client displays only one notification.

Declare tracker classes

To integrate the OpenCV tbm module, create a tracker that converts data between the integrations format and the format required by the module.

Create the following files in your project directory:

• object_tracker.h and object_tracker.cpp: Defines the ObjectTracker class using the OpenCV tbm module.

• object_tracker_utils.h and object_tracker_utils.cpp: Defines auxiliary utilities for object tracking.

Initialize the ObjectTracker

To initialize your tracking class within the device agent framework:

1. Include the object_tracker.h header in both device_agent.h and device_agent.cpp.

2. Add a private ObjectTracker member to the DeviceAgent class definition in device_agent.h:

std::unique_ptr<ObjectTracker> m_objectTracker;

3. Initialize m_objectTracker in the DeviceAgent class constructor inside device_agent.cpp:

DeviceAgent::DeviceAgent(
    const nx::sdk::IDeviceInfo* deviceInfo,
    std::filesystem::path pluginHomeDir):
    ConsumingDeviceAgent(deviceInfo, /*enableOutput*/ true),
    m_objectDetector(std::make_unique<ObjectDetector>(pluginHomeDir)),
    m_objectTracker(std::make_unique<ObjectTracker>())
{
}

Track objects across frames

To track objects, update the frame processing logic to pass detections to your tracker, and then assign the generated track IDs to your metadata packets.

1. Update DeviceAgent::processFrame() to run the tracker immediately after object detection:

DetectionList detections = m_objectDetector->run(frame);
detections = m_objectTracker->run(frame, detections);

2. Assign the trackId to the metadata packet inside the DeviceAgent::detectionsToObjectMetadataPacket() method:

Ptr<ObjectMetadataPacket> DeviceAgent::detectionsToObjectMetadataPacket(
    const DetectionList& detections,
    int64_t timestampUs)
{
    if (detections.empty())
        return nullptr;
        
    const auto objectMetadataPacket = makePtr<ObjectMetadataPacket>();
    for (const std::shared_ptr<Detection>& detection: detections)
    {
        const auto objectMetadata = makePtr<ObjectMetadata>();
        objectMetadata->setBoundingBox(detection->boundingBox);
        objectMetadata->setConfidence(detection->confidence);
        objectMetadata->setTrackId(detection->trackId);
        // ...

Handle frame size changes

Changes in video frame size (such as switching to a secondary stream) can cause errors in OpenCV. Reset the tracker when the frame size changes.

1. Add a method to check for dimension changes and reinitialize the tracker:

void DeviceAgent::reinitializeObjectTrackerOnFrameSizeChanges(const Frame& frame)
{
    const bool frameSizeUnset = m_previousFrameWidth == 0 && m_previousFrameHeight == 0;
    if (frameSizeUnset)
    {
        m_previousFrameWidth = frame.width;
        m_previousFrameHeight = frame.height;
        return;
    }
    const bool frameSizeChanged = frame.width != m_previousFrameWidth ||
        frame.height != m_previousFrameHeight;
    if (frameSizeChanged)
    {
        m_objectTracker = std::make_unique<ObjectTracker>();
        m_previousFrameWidth = frame.width;
        m_previousFrameHeight = frame.height;
    }
}

2. Invoke this method at the beginning of DeviceAgent::processFrame(), before performing object detection:

DeviceAgent::MetadataPacketList DeviceAgent::processFrame(
    const IUncompressedVideoFrame* videoFrame)
{
    const Frame frame(videoFrame, m_frameIndex);
    reinitializeObjectTrackerOnFrameSizeChanges(frame);
    // ...

Handle tracking errors

To differentiate tracking errors from detection errors, define dedicated exception classes and catch them during frame processing.

1. Define the tracking exception classes in exception.h:

class ObjectTrackerError: public Error { using Error::Error; };
class ObjectTrackingError: public ObjectTrackerError { using ObjectTrackerError::ObjectTrackerError; };

2. Add a catch block to DeviceAgent::processFrame() to handle tracking exceptions:

catch (const ObjectDetectionError& e)
{
    pushPluginDiagnosticEvent(
        IPluginDiagnosticEvent::Level::error,
        "Object detection error.",
        e.what());
    m_terminated = true;
}
catch (const ObjectTrackingError& e)
{
    pushPluginDiagnosticEvent(
        IPluginDiagnosticEvent::Level::error,
        "Object tracking error.",
        e.what());
    m_terminated = true;
}
return {};

You can now build your integration and verify that the system tracks objects correctly across frames.