Step 4. How to Add Object Tracking

For further reading we need full source code, because hereafter we are going to observe only the part related to the principal plugin, omitting OpenCV framework specifics. All the code examples are informational and descriptive and are not supposed to be copy/pasted anywhere. 

For full source code details see files in opencv_object_detection_analytics_plugin/step4/src/sample_company/vms_server_plugins/opencv_object_detection (see the section Getting the source code of the examples).

Note that at the moment, we process the first detection on the frame that the neural network is confident about and skip all other detections. There are several problems related to such functionality: 

• There can be multiple real-world objects and multiple detections on the frame. How can we determine which detection corresponds to which real-world object?
• Real-world objects can move, and consequently change their location in the picture from frame to frame causing new detections. Which new location corresponds to which objects?

To resolve these problems, we need to follow the possible changes in object coordinates in one frame and across several frames. This is known as tracking. We need to identify each object somehow, detect its changes and track them while keeping the association with the object identifier.

When a plugin identifies the similarity of several objects, it should conclude there is a track of the same object and somehow identify this track to distinguish it from other tracks.

This identifier is generally known as trackId and should be generated and assigned to all detections/objects of the track. This can be done with the ObjectMetadata.setTrackId (const Uuid &value) method before passing ObjectMetadata to the Server.

When the server receives several sequential ObjectMetadata with the same trackId it treats them as of the same object, so it appears in search results only once. In the same way, when the Desktop client receives several sequential ObjectMetadata with the same trackId it displays notification only once in the right panel.

Let’s learn how to implement tracking. For that, we use the tbm (tracking by matching) module from the OpenCV contrib library.

Declaring additional classes

The main idea of this code is implementing a tracker that would perform several operations:

1. Take a frame, detections, timestamp and frame index.
2. Convert them into the format suitable for the tbm module.
3. Pass the data to tbm tracker and receive the output.
4. Convert it back to the format used in our plugin. 

We create several new files for new classes, as we did previously: 

• ObjectTracker class utilizing the OpevCV tbm module.
  • object_tracker.h
  • object_tracker.cpp
• Several auxiliary utilities for object tracking.
  • object_tracker_utils.h
  • object_tracker_utils.cpp

See the complete code in the corresponding files of the step4 source code.

ObjectTracker initialization

Include the object_tracker.h header into device_agent.h and device_agent.cpp.

In the DeviceAgent class definition in the device_agent.h we add a new ObjectTracker member into the private section

std::unique_ptr<ObjectTracker> m_objectTracker;

In the DeviceAgent class constructor we initialize this ObjectTracker in a similar way as we did with ObjectDetector.

DeviceAgent::DeviceAgent(
    const nx::sdk::IDeviceInfo* deviceInfo,
    std::filesystem::path pluginHomeDir):
// Call the DeviceAgent helper class constructor telling it to verbosely report to stderr.
    ConsumingDeviceAgent(deviceInfo, /*enableOutput*/ true),
    m_objectDetector(std::make_unique<ObjectDetector>(pluginHomeDir)),
    m_objectTracker(std::make_unique<ObjectTracker>())
{
}

Tracking objects

Now we need our ObjectTracker to process detected objects. We perform detection in the DeviceAgent::processFrame(). So let’s make tracking take place just after detection and put the code there:

DetectionList detections = m_objectDetector->run(frame);
detections = m_objectTracker->run(frame, detections);

For how ObjectTracker::run() method is built see the object_detector.cpp and object_detector.h files.

When a plugin code identifies the similarity of several detections (objects), it should generate and assign the same trackId to those objects. It can be done with the ObjectMetadata.setTrackId (const Uuid &value) method before passing ObjectMetadata to the Server.

When the Desktop client receives several sequential ObjectMetadata with the same trackId it displays notification only once in the right panel.

Let’s ensure that trackId is correctly set. We do conversion of detections to ObjectMetadata in the detectionsToObjectMetadataPacket() method (see Detecting objects). We are going to add a line of code there.

Ptr<ObjectMetadataPacket> DeviceAgent::detectionsToObjectMetadataPacket(
    const DetectionList& detections,
    int64_t timestampUs)
{
    if (detections.empty())
        return nullptr;
    const auto objectMetadataPacket = makePtr<ObjectMetadataPacket>();
    for (const std::shared_ptr<Detection>& detection: detections)
    {
        const auto objectMetadata = makePtr<ObjectMetadata>();
        objectMetadata->setBoundingBox(detection->boundingBox);
        objectMetadata->setConfidence(detection->confidence);
        objectMetadata->setTrackId(detection->trackId);
...

Handling frame size changes

Because of the OpenCV specifics, we need to handle changes of the frame size (for example, when the stream has been changed to the secondary in the plugin settings on a device). For that we create a special method.

void DeviceAgent::reinitializeObjectTrackerOnFrameSizeChanges(const Frame& frame)
{
    const bool frameSizeUnset = m_previousFrameWidth == 0 && m_previousFrameHeight == 0;
    if (frameSizeUnset)
    {
        m_previousFrameWidth = frame.width;
        m_previousFrameHeight = frame.height;
        Return;
    }
    const bool frameSizeChanged = frame.width != m_previousFrameWidth ||
        frame.height != m_previousFrameHeight;
    if (frameSizeChanged)
    {
        m_objectTracker = std::make_unique<ObjectTracke r>();
        m_previousFrameWidth = frame.width;
        m_previousFrameHeight = frame.height;
    }
}

and invoke it in processFrame() before object detection.

DeviceAgent::MetadataPacketList DeviceAgent::processFrame(
    const IUncompressedVideoFrame* videoFrame)
{
    const Frame frame(videoFrame, m_frameIndex);
    reinitializeObjectTrackerOnFrameSizeChanges(frame);
...

Handling tracking errors

In order to differentiate errors, let’s define special exception classes for object tracking errors in exception.h.

class ObjectTrackerError: public Error { using Error::Error; };
class ObjectTrackingError: public ObjectTrackerError { using ObjectTrackerError::ObjectTrackerError; };

On an object tracking error, in ObjectTracker::run() we’ll throw the ObjectTrackingError exception, which is going to be caught in processFrame(), as well. Compare to Object detection error handling.

catch (const ObjectDetectionError& e)
{
    pushPluginDiagnosticEvent(
        IPluginDiagnosticEvent::Level::error,
        "Object detection error.",
        e.what());
    m_terminated = true;
}
catch (const ObjectTrackingError& e)
{
    pushPluginDiagnosticEvent(
        IPluginDiagnosticEvent::Level::error,
        "Object tracking error.",
        e.what());
    m_terminated = true;
}
return {};

That’s it. Let’s build our plugin and see how object tracking works.