Bounding boxes are not visible/working for custom meta data packets
AnsweredI am using external python API to receive coordinates and I am using Hello World plugin as my template.
Using this I am able to view rectangle box on client Application. since my application requires to draw multiple rectangles I implemented pullMetadataPackets using Stub Application plugin.
After modifying my plugin it looks like below,
pullMetadataPackets function
bool DeviceAgent::pullMetadataPackets(std::vector<IMetadataPacket*>* metadataPackets)
{
if((m_frameIndex%3)==0){
*metadataPackets = cookSomeObjects();
m_lastVideoFrameTimestampUs = 0;
nx::kit::utils::format("generated %d metadata packet(s)", metadataPackets->size());
// metadataPackets->push_back(generateObjectMetadataPacket().releasePtr());
}
return true; //< There were no errors while filling metadataPackets.
}
cookSomeObjects function
std::vector<IMetadataPacket*> DeviceAgent::cookSomeObjects()
{
std::unique_lock<std::mutex> lock(m_objectGenerationMutex);
std::vector<IMetadataPacket*> result;
if (m_lastVideoFrameTimestampUs == 0)
return result;
if (m_frameTimestampUsQueue.empty())
return result;
const auto metadataTimestampUs = m_frameTimestampUsQueue.front();
auto objectMetadataPacket = makePtr<ObjectMetadataPacket>();
objectMetadataPacket->setTimestampUs(metadataTimestampUs);
objectMetadataPacket->setDurationUs(0);
const microseconds delay(m_lastVideoFrameTimestampUs - metadataTimestampUs);
m_frameTimestampUsQueue.pop_front();
CURL *curl;
CURLcode res;
std::string readBuffer;
std::string body = "path=/var/www/html/images/"+std::to_string(m_lastVideoFrameTimestampUs)+".jpg";
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "localhost:5000/process");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
auto j = json::parse(readBuffer);
for (auto& context : j) {
float a = context["x1"];
float b = context["y1"];
float c = context["height"];
float d = context["width"];
auto objectMetadata = makePtr<ObjectMetadata>();
objectMetadata->setTypeId(kHelloWorldObjectType);
objectMetadata->setTrackId(m_trackId);
objectMetadata->setBoundingBox(Rect(a, b, c, d));
objectMetadataPacket->addItem(objectMetadata.get());
}
result.push_back(objectMetadataPacket.releasePtr());
return result;
}
The above code is not drawing any of the rectangles. It seems something is missing in cooksomeobjects Function. I don't know how to proceed further. Is there anything missing or not handled properly in the above code? any hint would be appreciable.
-
Official comment
Mohamed had found the root cause of the problem and described it here:
Comment actions -
Hello Mohamed,
1. It order the bounding box to be displayed in the live stream on the scene the metadata has to returned in lest then 2 seconds delay. Are you sure your code (curl query over HTTP, response processing etc.) does not introduce the delay more then 2 seconds?
2. Does your query return any content to the readBuffer ? Does j has a value?
Try to dump to log the objectMetadata just before
objectMetadataPacket->addItem(objectMetadata.get());
Does it look like what you expect it to be?
3. Where is m_trackId initialized? Does it have the same value over frames (i.e. pullMetadataPackets calls)?
-
Hello Andrey,
After a few modifications on the existing code I am able to view a single bounding box on client Application. Still multiples are not visible even though I am passing multiple coordinates.
1. It order the bounding box to be displayed in the live stream on the scene the metadata has to returned in lest then 2 seconds delay. Are you sure your code (curl query over HTTP, response processing etc.) does not introduce the delay more then 2 seconds?
Yes, it returns less than a sec. So, this is not the problem.
2. Does your query return any content to the readBuffer ? Does j has a value?
yes, it readBuffer has proper value. Sample Response of J.
[{"height":0.10374343395233154,"label":"car","width":0.13737773895263672,"x1":0.4747169017791748,"y1":0.27557021379470825},{"height":0.1520291566848755,"label":"car","width":0.23306196182966232,"x1":0.0436464324593544,"y1":0.4737709164619446},{"height":0.05128565430641174,"label":"car","width":0.08260226249694824,"x1":0.5227476954460144,"y1":0.1012464314699173},{"height":0.2633265256881714,"label":"car","width":0.2628181278705597,"x1":0.26807519793510437,"y1":0.6712638735771179},{"height":0.1066703349351883,"label":"car","width":0.1351754069328308,"x1":0.3640907108783722,"y1":0.19261829555034637},{"height":0.13967299461364746,"label":"bike","width":0.09549647569656372,"x1":0.22761660814285278,"y1":0.5885671973228455},{"height":0.06679290533065796,"label":"car","width":0.07677030563354492,"x1":0.7689603567123413,"y1":0.15423545241355896},{"height":0.08153106272220612,"label":"car","width":0.09254050254821777,"x1":0.6379277110099792,"y1":0.19936798512935638},{"height":0.13518506288528442,"label":"car","width":0.11784732341766357,"x1":0.6905719637870789,"y1":0.34653833508491516}]
NX_PRINT<<objectMetadata;
objectMetadataPacket->addItem(objectMetadata.get());>>> The above snippet prints 1. i.e., for each iteration of object in J, it prints 1. (I think it's desired result)
3. Where is m_trackId initialized? Does it have the same value over frames (i.e. pullMetadataPackets calls)?
m_trackId is initialized in generateEventMetadataPacketm_trackId = nx::sdk::UuidHelper::randomUuid();
It has the below value and it's same value over frames.
C47F7161-1D99-457F-B86E-000B09149811
Apart from this, When I print `objectMetadataPacket` at end of each object J iteration completion it has only 1 value. i think this is the issue. objectMetadataPacket is holding single value, instead of appending with previous values.
`NX_PRINT<<objectMetadataPacket;
result.push_back(objectMetadataPacket.releasePtr());`this is a part of log which illustrates the current issue.
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] processMetadataPackets(): Producing 1 metadata packet(s).
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] logMetadataPacketIfNeeded(): Object metadata packet #0 contains 8 item(s).
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] doPushDataPacket() END
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] doPushDataPacket() BEGIN
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] doPushDataPacket() END
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] doPushDataPacket() BEGIN
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] doPushDataPacket() END
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] doPushDataPacket() BEGIN[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] ----object Metadata---
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] =====objectMetadataPacket===
[sample_analytics_plugin_device_{e3e9a385-7fe0-3ba5-5482-a86cde7faf48}] 1
In the above log it says producing Object metadata packet #0 contains 8 item(s) as I expected. And it's pushing each object meta data into objectMetadataPacket one by one by iterating j. At the end of loop, objectMetadataPacket should contain 8 items. But, it holds only 1. I think this is the reason why I am getting only one rectangle box. If you need any other information please ask me.My current CookSomeObjects function looks like below,
std::vector<IMetadataPacket*> DeviceAgent::cookSomeObjects()
{std::vector<IMetadataPacket*> result;
auto objectMetadataPacket = makePtr<ObjectMetadataPacket>();
objectMetadataPacket->setTimestampUs(m_lastVideoFrameTimestampUs);
objectMetadataPacket->setDurationUs(0);
CURL *curl;
CURLcode res;
std::string readBuffer;
std::string body = "path=/var/www/html/images/"+std::to_string(m_lastVideoFrameTimestampUs)+".jpg";curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "localhost:5000/process");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}auto j = json::parse(readBuffer);
for (auto& context : j) {
float a = context["x1"];
float b = context["y1"];
float c = context["height"];
float d = context["width"];
std::string label = context["label"];auto objectMetadata = makePtr<ObjectMetadata>();
if(label=="car"){
objectMetadata->setTypeId(kCarObjectType);
}
else if(label=="truck"){
objectMetadata->setTypeId(kTruckObjectType);
}
else if(label=="bike"){
objectMetadata->setTypeId(kBikeObjectType);
}
objectMetadata->setTrackId(m_trackId);
objectMetadata->setBoundingBox(Rect(a, b, c, d));objectMetadataPacket->addItem(objectMetadata.get());
}
result.push_back(objectMetadataPacket.releasePtr());
return result;}
Please sign in to leave a comment.
Comments
3 comments