Can we annotate our AI output that's ran by our server using NX Meta?
AnsweredGreetings people,
I'd like to ask whether is it possible to write or annotate our AI object detection & tracking output and send it to NX Meta such that on NX Client, the user can see the bounding box, tracker id, score, etc. The details is that let say we have a video stream input rtsp, it will be consumed by both NX Server and our AI Engine, thus at the same frame, our AI Engine push the object detection metadata and let the NX Meta write an overlay for the bounding box on the streamed video.
This is the sample of the metadata
{
"objects": [
{
"name": "person",
"class": 6,
"confidence": 0.9294909834861755,
"box": [
1054.4952392578125,
201.50885009765625,
1171.42822265625,
516.9838256835938
],
"tracker_id": 1,
"milvus_id": 3
},
{
"name": "person",
"class": 6,
"confidence": 0.9289925694465637,
"box": [
1286.068115234375,
97.57745361328125,
1362.787109375,
338.76837158203125
],
"tracker_id": 2,
"milvus_id": 2
},
{
"name": "person",
"class": 6,
"confidence": 0.9228980541229248,
"box": [
653.0865478515625,
295.3555908203125,
767.2735595703125,
498.39788818359375
],
"tracker_id": 3,
"milvus_id": 7
},
{
"name": "person",
"class": 6,
"confidence": 0.9126408100128174,
"box": [
1190.5106201171875,
170.9093780517578,
1276.2149658203125,
453.71368408203125
],
"tracker_id": 4,
"milvus_id": 6
}
]
}
Thank you for the support.
-
Hello,
The task you're describing is feasible.
At the moment the only way to deliver metadata (bounding box, tracker id, score, etc) to the Server is developing an analytics plugin with the Metadata SDK.
In your case, the plugin would accept metadata sent by your AI engine, transform to the format the Server understands and would push it tho the Server.
The recommended pipeline would be the following.
1. The Server consumes a rtsp stream, your AI Engine consumes the stream from the Server. The reason is the frames timestamps synchronization. By default, the Server does not trust timestamps of the video source and applies it own based on the so-called System time synchronized over all the Servers in the System. That's the recommended way to deal with time and timestamp.
2. AI Engine processes the video stream and sends metadata to the plugin loaded to the Server. In addition to the parameter you mentioned the timestamp is the mandatory one in order the plugin is capable of notifying the Server what frame the metadata belongs to. Since timestamps are in sync (see step 1) the metadata are bound to the proper frame.
3. The plugin pushes the metadata to the Server.
Here are several links about plugin subsystem architecture and plugin development.
https://meta.nxvms.com/docs/developers/knowledgebase/195-1-integration-capabilities--what-to-expect
Please sign in to leave a comment.
Comments
1 comment