Missing attributes in object list
AnsweredEvery metadata object has two attributes non-conditionally assigned; sometimes no attributes appear under the object in the object list. Is this expected behaviour?
The attributes are seen on the live replay.
Every metadatapacket adds attributes with the code:
auto bBox = getBoundingBox(detection);
objectMetadata->setBoundingBox(bBox);
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", std::to_string(detection.blurriness)));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string(detection.confidence)));
objectMetadataPacket->addItem(objectMetadata.get());
metadataPackets->push_back(objectMetadataPacket);
Nx Witness client/server version is 4.2.0.32840
-
Hello David Brown,
Could you please do the following?
Port a part of your code to the sample_analytics_plugin of the Metadata SDK and try to reproduce the issue by generating "hello world" objects with additional attributes with random values.
If the issue persists, please share this code with us.
JIRA-VMS-30482
-
Hi Andrey,
Thanks for the reply. I modified the sample_analytics_plugin, similar to how you mentioned, and while I'm not currently seeing attributes missing, I am seeing numerous best shot images that were previously added by the other plugin (the other plugin is disabled and the dll file removed from the plugins folder) being applied against the "Hello, World" objects. Some of these images would be from >5 days ago.
As I understand it, a user-provided best shot image should not be applied to a object in the list unless its trackId is matched. UUIDs using being generated here using nx::sdk::UuidHelper::randomUuid().
Applying and removing a filter (e.g. time=last day filter) will remove some of those images from the objects in the object list, and apply new best shot images to other objects. The same issue occurs using nx witness client on a different machine, so it would appear to be an issue at the server.
Do you have any insight into how/why this is happening?
The code is based on the sample_analytics_plugin, with the following functions modified:
Ptr<IMetadataPacket> DeviceAgent::generateEventMetadataPacket()
{
// Generate event every kTrackFrameCount'th frame.
if (m_frameIndex % 16 != 0)
return nullptr;
// EventMetadataPacket contains arbitrary number of EventMetadata.
const auto eventMetadataPacket = makePtr<EventMetadataPacket>();
// Bind event metadata packet to the last video frame using a timestamp.
eventMetadataPacket->setTimestampUs(m_lastVideoFrameTimestampUs);
// Zero duration means that the event is not sustained, but momental.
eventMetadataPacket->setDurationUs(0);
// EventMetadata contains an information about event.
const auto eventMetadata = makePtr<EventMetadata>();
// Set all required fields.
eventMetadata->setTypeId(kNewTrackEventType);
eventMetadata->setIsActive(true);
eventMetadata->setCaption("New sample plugin track started");
eventMetadata->setDescription("New track #" + std::to_string(m_trackIndex) + " started");
eventMetadataPacket->addItem(eventMetadata.get());
// Generate index and track id for the next track.
++m_trackIndex;
// added: int DeviceAgent::updateCounter = 0;
// added: std::vector<nx::sdk::Uuid> DeviceAgent::trackIds;
if (trackIds.size() == 0)
{
static constexpr int numTracks = 15;
trackIds.resize(numTracks);
for(int i = 0; i < trackIds.size(); i++)
trackIds[i] = nx::sdk::UuidHelper::randomUuid();
}
if(++updateCounter == 50)
{
for(int i = 0; i < trackIds.size(); i++)
{
trackIds[rand() % trackIds.size()] = nx::sdk::UuidHelper::randomUuid();
}
updateCounter = 0;
}
m_trackId = trackIds[rand() % trackIds.size()];
return eventMetadataPacket;
}
Ptr<IMetadataPacket> DeviceAgent::generateObjectMetadataPacket()
{
// ObjectMetadataPacket contains arbitrary number of ObjectMetadata.
const auto objectMetadataPacket = makePtr<ObjectMetadataPacket>();
// Bind the object metadata to the last video frame using a timestamp.
objectMetadataPacket->setTimestampUs(m_lastVideoFrameTimestampUs);
objectMetadataPacket->setDurationUs(30000);
// ObjectMetadata contains information about an object on the frame.
const auto objectMetadata = makePtr<ObjectMetadata>();
// Set all required fields.
objectMetadata->setTypeId(kHelloWorldObjectType);
objectMetadata->setTrackId(m_trackId);
// Calculate bounding box coordinates each frame so that it moves from the top left corner
// to the bottom right corner during kTrackFrameCount frames.
static constexpr float d = 0.5F / kTrackFrameCount;
static constexpr float width = 0.5F;
static constexpr float height = 0.5F;
const int frameIndexInsideTrack = m_frameIndex % kTrackFrameCount;
const float x = d * frameIndexInsideTrack;
const float y = d * frameIndexInsideTrack;
objectMetadata->setBoundingBox(Rect(x, y, width, height));
constexpr int mod = 30;
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", std::to_string((float)(rand() % mod) / mod)));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string((float)(rand() % mod) / mod)));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "trackId", nx::sdk::UuidHelper::toStdString(m_trackId)));
objectMetadataPacket->addItem(objectMetadata.get());
return objectMetadataPacket;
}And two members added to the DeviceAgent.h file:
int updateCounter = 0;
std::vector<nx::sdk::Uuid> trackIds; -
Hi David,
The root cause of the issue is duplicated trackIds as it's described in the "Object Preview Image Is Wrong" of this guide https://support.networkoptix.com/hc/en-us/articles/1500006332441-Troubleshooting-Analytics-Issues-for-Cameras-and-Plugins
The question is - why IDs are duplicated?
I have not yet managed to reproduce the issue with the code you've provided, since the issue appears in about 5 days, as you have noted.
-
Hi Andrey,
Thanks for following up. I can't imagine why the trackIds would be duplicated, and at such a high rate, because they are created using nx::sdk::UuidHelper::randomUuid(); they should be different to UUIDs created by our system.
The behaviour has since ceased, for no known reason, though our other plugin has been disabled, so no new bestShot jpg images were pushed into the system.
-
Hi Andrey,
Regarding the original issue, which still exists, I've looked at the desktop logs and can see the metadata against the trackId of interest.
For some objects, the metadata shows up intially, and will disappear for <1 second when the "last day" filter is selected or unselected, before reappearing. Then a new track started and the metadata is no longer visible.
-
Hi Andrey,
I have managed to reproduce this issue using the sdk sample plugin, modified to add best shot images. You can see the issue manifesting in this video:
https://file-store.surge.sh/Disappearing-attributes.mp4
The attributes will disappear, and sometimes the best shot image is not applied (but will appear when refreshing the list by changing the filter).
The files changed from the stock sample plugin are here:
https://file-store.surge.sh/device_agent.h
https://file-store.surge.sh/device_agent.cpp
I make the best shot jpg files using this C# program, placed on the nx server C:\ path:
https://file-store.surge.sh/MakeJpeg.zip
This simply makes and saves an image containing three lines of text.
Here's another video showing the incorrect images being drawn against the HelloWorld objects (which shouldn't have any jpg assigned to it at all). The black images are from a previous run of this plugin, and shouldn't be showing up interleaved with the current green images. This is using the same code as above. Video here:
https://file-store.surge.sh/Erroneous-best-shot-images.mp4
Regards,
-
Hello David,
Thank you so much for a such detailed explanation.
Our developers said the notification panel code changed a lot since 4.2.
I tried to reproduce the issue with your code in NxMeta 5.0 R4. Did not succeed.
Could you, please, recheck on your side if the issue persists with NxMeta 5.0 R4?
-
Hi Andrey,
I just tried the plugin on Meta 5.0 R4, and the issue seen in 4.2 persists. The attributes disappear from the objects list when the filter is changed, as evidenced here:
https://file-store.surge.sh/Disappearing-attributes-meta-v5.mp4
There are no other custom plugins installed; just the modified sample analytics plugin provided (code is above).
I have not yet seen the issue with the randomly-substituted best-shot-images, on v5.0, but that can take some time before manifesting.
-
Hello David,
We have some fixes made in the GUI application.
And here are notes and detailed debug information from our developer on the plugin behavior.
====
There is some bug in the plugin. I've added some debug logs to investigate it:
```
2022-01-14 20:28:43.213 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c31058): Insert trackId={21000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
2022-01-14 20:28:43.215 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c31058): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:28:49.812 4898 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2e658): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:28:53.080 719c WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56820ebb8): Insert trackId={25000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
2022-01-14 20:28:54.812 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2eb58): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:28:59.813 719c WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2e558): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:29:04.542 719c WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2e858): Insert trackId={29000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
2022-01-14 20:29:06.480 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c33358): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:29:09.607 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f5682136b8): Insert trackId={2d000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
2022-01-14 20:29:11.542 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a565758): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:29:16.544 4898 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a565658): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:29:19.742 4898 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2c158): Insert trackId={31000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
2022-01-14 20:29:23.145 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a563758): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
2022-01-14 20:29:24.811 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a563458): Insert trackId={35000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
```So, the plugin goes:
1. It creates new tracks every several seconds with new trackId. And it never fills attribute "Frame Index" for them.
2. It updates attribute "Frame Index" in the track with fixed Id "{29af1ad5-ef2d-406f-bfe0-27533704ca00}".So, a bug looks like the plugin puts this attribute to the wrong track.
-
Hi Andrey,
Thanks for following up on this. What you describe is the expected behaviour of the plugin (one object with constant track ID and "Frame Index" attribute, and ephemeral objects with different track IDs and attributes).
I did notice, however, a bug in the above plugin test code that erroneously sets an attribute with a blank name and blank value. This appears to cause _all_ attributes for that object to disappear, as seen in the video. With the blank attribute removed, Nx meta appears to display the attributes correctly. It's surprising that nx meta should hide all valid attributes instead of just rejecting invalid ones.
I will still try to test the other issue regarding invalid best-shot images.
-
Hello David,
I did notice, however, a bug in the above plugin test code that erroneously sets an attribute with a blank name and blank value.
Could you please specify what attribute in particular do you mean?
In your code, I see three attributes set with the explicit names, that could not be blank.
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", std::to_string((float)(rand() % mod) / mod)));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string((float)(rand() % mod) / mod)));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "trackId", nx::sdk::UuidHelper::toStdString(m_trackId)));
objectMetadataPacket->addItem(objectMetadata.get());In other words, could you please give me instructions to reproduce your scenario, namely this one
This appears to cause _all_ attributes for that object to disappear, as seen in the video
-
Andrey,
Could you please specify what attribute in particular do you mean?
In your code, I see three attributes set with the explicit names, that could not be blank.
case 1: metadataPackets->push_back(generateObjectMetadataPacket(switchId, kHelloWorldObjectType, 0.75f, 0.35f, {{"state", std::to_string(switchIndex)}, {}}).releasePtr()); break;
You've probably found it, but I put an empty {} in the attributes map above.
-
Hello David,
It seems, we have two separate phenomena to consider.
1. The issue that GUI does not display attributes as it's shown on your video.
https://file-store.surge.sh/Disappearing-attributes-meta-v5.mp4
The root cause for that is track id confusion, as described above. In simple words, attributes were assigned to wrong track id and thus were not displayed for the object they were expected to be shown for.
2. Assumption about possible cause for 1st point.
test code that erroneously sets an attribute with a blank name and blank value. This appears to cause _all_ attributes for that object to disappear, as seen in the video.
That assumption was not confirmed in the experiments.
#1
I took the sample_analytics_plugin and modified the code of the DeviceAgent::generateObjectMetadataPacket() like this:
objectMetadata->setBoundingBox(Rect(x, y, width, height));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", ""));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "", ""));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string(0.77)));
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "trackId", nx::sdk::UuidHelper::toStdString(m_trackId)));
objectMetadataPacket->addItem(objectMetadata.get());
return objectMetadataPacket;
}Expected result:
If the assumption is correct, no attributes are displayed, i.e. all attributes disappear in the right panel.
Actual result:
All the attributes are displayed
#2
I took the stub_analytics_plugin and modified the code of the samples/stub_analytics_plugin/src/nx/vms_server_plugins/analytics/stub/deprecated_object_detection/device_agent.cpp
static Ptr<IObjectMetadata> makeObjectMetadata(const AbstractObject* object)
{
auto objectMetadata = makePtr<ObjectMetadata>();
objectMetadata->setTypeId(object->typeId());
objectMetadata->setTrackId(object->id());
const auto position = object->position();
const auto size = object->size();
objectMetadata->setBoundingBox(Rect(position.x, position.y, size.width, size.height));
objectMetadata->addAttributes(object->attributes());
objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "", ""));
return objectMetadata;
}Expected result:
If the assumption is correct, no attributes are displayed, i.e. all attributes disappear in the right panel.
Actual result:
All the attributes are displayed.
-
Hi Andrey,
Thank you for investigating. I don't understand how you cannot reproduce it. Are you sure you are toggling the time filter setting in the objects list pane?
I applied the 4 lines you prescribed to the DeviceAgent::generateObjectMetadataPacket() of a fresh copy of the sample_analytics_plugin. This causes the all attributes to disappear, after toggling the object time filter between, e.g. Any Time and Last Day, as shown in the previous video.
If I include either of the two lines (from your example) containing an empty string, this bug manifests. If I only keep the lower two lines (i.e. those with non-empty values), then the attributes remain (as is proper). This bug is consistently reproducible, and consistently fixable (when removing the empty attribute value).
I'm using the metavms-metadata_sdk-4.2.0.32836-universal.zip, and I just tested on a local installation of nx meta 5.0.0.33871 R4. I have the plugin dll copied into the plugins folder.
-
Hi Andrey,
Is there any update on this issue? I'm more concerned with old best shot images being applied to incorrect tracks, but wonder if these two issues are related.
Also, regarding track confusion reported by the developer, I'd like to reiterate that it was not track confusion displayed in the logs, but was expected behaviour. The UPDATE entries are for an eternal track that changes attributes each frame, and the INSERT entries are for ephemeral tracks that have non-changing attribute values. The latter tracks are easily identified the first 8 characters of the track ID. Hopefully this makes sense; if not, there's an image here that might describes it better: https://file-store.surge.sh/attributes-insert-update.png
Cheers
Please sign in to leave a comment.
Comments
25 comments