Missing attributes in object list

Answered

Comments

25 comments

  • Avatar
    Andrey Terentyev

    Hello David Brown,

    Could you please do the following?

    Port a part of your code to the sample_analytics_plugin of the Metadata SDK and try to reproduce the issue by generating "hello world" objects with additional attributes with random values.

    If the issue persists, please share this code with us.

    JIRA-VMS-30482

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Thanks for the reply. I modified the sample_analytics_plugin, similar to how you mentioned, and while I'm not currently seeing attributes missing, I am seeing numerous best shot images that were previously added by the other plugin (the other plugin is disabled and the dll file removed from the plugins folder) being applied against the "Hello, World" objects. Some of these images would be from >5 days ago.

    As I understand it, a user-provided best shot image should not be applied to a object in the list unless its trackId is matched. UUIDs using being generated here using nx::sdk::UuidHelper::randomUuid().

    Applying and removing a filter (e.g. time=last day filter) will remove some of those images from the objects in the object list, and apply new best shot images to other objects. The same issue occurs using nx witness client on a different machine, so it would appear to be an issue at the server.

    Do you have any insight into how/why this is happening?

     

    The code is based on the sample_analytics_plugin, with the following functions modified:

    Ptr<IMetadataPacket> DeviceAgent::generateEventMetadataPacket()
    {
      // Generate event every kTrackFrameCount'th frame.
      if (m_frameIndex % 16 != 0)
          return nullptr;
      // EventMetadataPacket contains arbitrary number of EventMetadata.
      const auto eventMetadataPacket = makePtr<EventMetadataPacket>();
      // Bind event metadata packet to the last video frame using a timestamp.
      eventMetadataPacket->setTimestampUs(m_lastVideoFrameTimestampUs);
      // Zero duration means that the event is not sustained, but momental.
      eventMetadataPacket->setDurationUs(0);
      // EventMetadata contains an information about event.
      const auto eventMetadata = makePtr<EventMetadata>();
      // Set all required fields.
      eventMetadata->setTypeId(kNewTrackEventType);
      eventMetadata->setIsActive(true);
      eventMetadata->setCaption("New sample plugin track started");
      eventMetadata->setDescription("New track #" + std::to_string(m_trackIndex) + " started");
      eventMetadataPacket->addItem(eventMetadata.get());
      // Generate index and track id for the next track.
      ++m_trackIndex;
      // added: int DeviceAgent::updateCounter = 0;
      // added: std::vector<nx::sdk::Uuid> DeviceAgent::trackIds;
      if (trackIds.size() == 0)
      {
          static constexpr int numTracks = 15;
          trackIds.resize(numTracks);
          for(int i = 0; i < trackIds.size(); i++)
              trackIds[i] = nx::sdk::UuidHelper::randomUuid();
      }
     
      if(++updateCounter == 50)
      {
          for(int i = 0; i < trackIds.size(); i++)
          {
              trackIds[rand() % trackIds.size()] = nx::sdk::UuidHelper::randomUuid();
          }
          updateCounter = 0;
      }
      m_trackId = trackIds[rand() % trackIds.size()];
      return eventMetadataPacket;
    }
    Ptr<IMetadataPacket> DeviceAgent::generateObjectMetadataPacket()
    {
      // ObjectMetadataPacket contains arbitrary number of ObjectMetadata.
      const auto objectMetadataPacket = makePtr<ObjectMetadataPacket>();
      // Bind the object metadata to the last video frame using a timestamp.
      objectMetadataPacket->setTimestampUs(m_lastVideoFrameTimestampUs);
      objectMetadataPacket->setDurationUs(30000);
      // ObjectMetadata contains information about an object on the frame.
      const auto objectMetadata = makePtr<ObjectMetadata>();
      // Set all required fields.
      objectMetadata->setTypeId(kHelloWorldObjectType);
      objectMetadata->setTrackId(m_trackId);
      // Calculate bounding box coordinates each frame so that it moves from the top left corner
      // to the bottom right corner during kTrackFrameCount frames.
      static constexpr float d = 0.5F / kTrackFrameCount;
      static constexpr float width = 0.5F;
      static constexpr float height = 0.5F;
      const int frameIndexInsideTrack = m_frameIndex % kTrackFrameCount;
      const float x = d * frameIndexInsideTrack;
      const float y = d * frameIndexInsideTrack;
      objectMetadata->setBoundingBox(Rect(x, y, width, height));
      constexpr int mod = 30;
      objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", std::to_string((float)(rand() % mod) / mod)));
      objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string((float)(rand() % mod) / mod)));
      objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "trackId", nx::sdk::UuidHelper::toStdString(m_trackId)));
      objectMetadataPacket->addItem(objectMetadata.get());
      return objectMetadataPacket;
    }

    And two members added to the DeviceAgent.h file:

        int updateCounter = 0;
        std::vector<nx::sdk::Uuid> trackIds;
    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hi David,

    Thank you for your detailed feedback. I'll check it.

    Which OS did you build the plugin for?

    JIRA-VMS-30482

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey, 

    Thanks for the reply. I'm building on Windows 10. I'd be very interested in any findings.

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hi David,

    The root cause of the issue is duplicated trackIds as it's described in the "Object Preview Image Is Wrong" of this guide https://support.networkoptix.com/hc/en-us/articles/1500006332441-Troubleshooting-Analytics-Issues-for-Cameras-and-Plugins

    The question is - why IDs are duplicated?

    I have not yet managed to reproduce the issue with the code you've provided, since the issue appears in about 5 days, as you have noted.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Thanks for following up. I can't imagine why the trackIds would be duplicated, and at such a high rate, because they are created using nx::sdk::UuidHelper::randomUuid(); they should be different to UUIDs created by our system.

    The behaviour has since ceased, for no known reason, though our other plugin has been disabled, so no new bestShot jpg images were pushed into the system.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Regarding the original issue, which still exists, I've looked at the desktop logs and can see the metadata against the trackId of interest.

    For some objects, the metadata shows up intially, and will disappear for <1 second when the "last day" filter is selected or unselected, before reappearing. Then a new track started and the metadata is no longer visible. 

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    I have managed to reproduce this issue using the sdk sample plugin, modified to add best shot images. You can see the issue manifesting in this video:

                       https://file-store.surge.sh/Disappearing-attributes.mp4

    The attributes will disappear, and sometimes the best shot image is not applied (but will appear when refreshing the list by changing the filter). 

     

    The files changed from the stock sample plugin are here:

            https://file-store.surge.sh/device_agent.h

            https://file-store.surge.sh/device_agent.cpp 

     

    I make the best shot jpg files using this C# program, placed on the nx server C:\ path:

            https://file-store.surge.sh/MakeJpeg.zip

    This simply makes and saves an image containing three lines of text.

     

    Here's another video showing the incorrect images being drawn against the HelloWorld objects (which shouldn't have any jpg assigned to it at all). The black images are from a previous run of this plugin, and shouldn't be showing up interleaved with the current green images.  This is using the same code as above. Video here:

               https://file-store.surge.sh/Erroneous-best-shot-images.mp4 

     

     

    Regards,

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David,

    Thank you so much for a such detailed explanation.

    Our developers said the notification panel code changed a lot since 4.2.

    I tried to reproduce the issue with your code in NxMeta 5.0 R4. Did not succeed.

    Could you, please, recheck on your side if the issue persists with NxMeta 5.0 R4?

     

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Thanks for looking into this. The issue seems to occur routinely on 4.2.

    I will try with 5.0. 

    Regards,

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    I just tried the plugin on Meta 5.0 R4, and the issue seen in 4.2 persists. The attributes disappear from the objects list when the filter is changed, as evidenced here:

           https://file-store.surge.sh/Disappearing-attributes-meta-v5.mp4 

    There are no other custom plugins installed; just the modified sample analytics plugin provided (code is above).

    I have not yet seen the issue with the randomly-substituted best-shot-images, on v5.0, but that can take some time before manifesting.

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David,

    Thank you very much for your findings.

    I reported a bug and passed the issue to our devs.

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David,

    We have some fixes made in the GUI application.

    And here are notes and detailed debug information from our developer on the plugin behavior.

    ====

    There is some bug in the plugin. I've added some debug logs to investigate it:

    ```
    2022-01-14 20:28:43.213 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c31058): Insert trackId={21000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
    2022-01-14 20:28:43.215 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c31058): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:28:49.812 4898 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2e658): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:28:53.080 719c WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56820ebb8): Insert trackId={25000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
    2022-01-14 20:28:54.812 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2eb58): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:28:59.813 719c WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2e558): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:29:04.542 719c WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2e858): Insert trackId={29000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
    2022-01-14 20:29:06.480 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c33358): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:29:09.607 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f5682136b8): Insert trackId={2d000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
    2022-01-14 20:29:11.542 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a565758): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:29:16.544 4898 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a565658): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:29:19.742 4898 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f569c2c158): Insert trackId={31000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
    2022-01-14 20:29:23.145 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a563758): UPDATE trackId={29af1ad5-ef2d-406f-bfe0-27533704ca00}, hasFrameAttr=true
    2022-01-14 20:29:24.811 7d94 WARNING nx::analytics::db::ObjectTrackDataSaver(0x1f56a563458): Insert trackId={35000000-11b9-4af3-9b1a-718e296c3789}, hasFrameAttr=false
    ```

    So, the plugin goes:
    1. It creates new tracks every several seconds with new trackId. And it never fills attribute "Frame Index" for them.
    2. It updates attribute "Frame Index" in the track with fixed Id "{29af1ad5-ef2d-406f-bfe0-27533704ca00}".

    So, a bug looks like the plugin puts this attribute to the wrong track.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Thanks for following up on this. What you describe is the expected behaviour of the plugin (one object with constant track ID and "Frame Index" attribute, and ephemeral objects with different track IDs and attributes).

    I did notice, however, a bug in the above plugin test code that erroneously sets an attribute with a blank name and blank value. This appears to cause _all_ attributes for that object to disappear, as seen in the video. With the blank attribute removed, Nx meta appears to display the attributes correctly. It's surprising that nx meta should hide all valid attributes instead of just rejecting invalid ones.

    I will still try to test the other issue regarding invalid best-shot images.

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David,

    I did notice, however, a bug in the above plugin test code that erroneously sets an attribute with a blank name and blank value.

    Could you please specify what attribute in particular do you mean?

    In your code, I see three attributes set with the explicit names, that could not be blank.

    objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", std::to_string((float)(rand() % mod) / mod)));
    objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string((float)(rand() % mod) / mod)));
    objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "trackId", nx::sdk::UuidHelper::toStdString(m_trackId)));
    objectMetadataPacket->addItem(objectMetadata.get());

    In other words, could you please give me instructions to reproduce your scenario, namely this one

    This appears to cause _all_ attributes for that object to disappear, as seen in the video

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    David,

    I found the proper code. Will try to reproduce your scenario with empty attribute name and value.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Andrey,

    Could you please specify what attribute in particular do you mean?

    In your code, I see three attributes set with the explicit names, that could not be blank.

    case 1: 
      metadataPackets->push_back(generateObjectMetadataPacket(switchId, kHelloWorldObjectType, 0.75f, 0.35f, {{"state", std::to_string(switchIndex)}, {}}).releasePtr());
      break;

    You've probably found it, but I put an empty {} in the attributes map above.

     

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David,

    It seems, we have two separate phenomena to consider.

    1. The issue that GUI does not display attributes as it's shown on your video.

    https://file-store.surge.sh/Disappearing-attributes-meta-v5.mp4 

    The root cause for that is track id confusion, as described above. In simple words, attributes were assigned to wrong track id and thus were not displayed for the object they were expected to be shown for.

    2. Assumption about possible cause for 1st point.

    test code that erroneously sets an attribute with a blank name and blank value. This appears to cause _all_ attributes for that object to disappear, as seen in the video.

    That assumption was not confirmed in the experiments.

    #1

    I took the sample_analytics_plugin and modified the code of the DeviceAgent::generateObjectMetadataPacket() like this:

        objectMetadata->setBoundingBox(Rect(x, y, width, height));
        objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Bluriness", ""));
        objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "", ""));
        objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "Confidence", std::to_string(0.77)));
        objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "trackId", nx::sdk::UuidHelper::toStdString(m_trackId)));
        objectMetadataPacket->addItem(objectMetadata.get());

      return objectMetadataPacket;
    }

    Expected result:

    If the assumption is correct, no attributes are displayed, i.e. all attributes disappear in the right panel.

    Actual result:

    All the attributes are displayed

    #2

    I took the stub_analytics_plugin and modified the code of the samples/stub_analytics_plugin/src/nx/vms_server_plugins/analytics/stub/deprecated_object_detection/device_agent.cpp

    static Ptr<IObjectMetadata> makeObjectMetadata(const AbstractObject* object)
    {
        auto objectMetadata = makePtr<ObjectMetadata>();
        objectMetadata->setTypeId(object->typeId());
        objectMetadata->setTrackId(object->id());
        const auto position = object->position();
        const auto size = object->size();
        objectMetadata->setBoundingBox(Rect(position.x, position.y, size.width, size.height));
        objectMetadata->addAttributes(object->attributes());
        objectMetadata->addAttribute(nx::sdk::makePtr<Attribute>(IAttribute::Type::string, "", ""));
        return objectMetadata;
    }

    Expected result:

    If the assumption is correct, no attributes are displayed, i.e. all attributes disappear in the right panel.

    Actual result:

    All the attributes are displayed.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Thank you for investigating. I don't understand how you cannot reproduce it. Are you sure you are toggling the time filter setting in the objects list pane? 

    I applied the 4 lines you prescribed to the DeviceAgent::generateObjectMetadataPacket() of a fresh copy of the sample_analytics_plugin. This causes the all attributes to disappear, after toggling the object time filter between, e.g. Any Time and Last Day, as shown in the previous video. 

    If I include either of the two lines (from your example) containing an empty string, this bug manifests. If I only keep the lower two lines (i.e. those with non-empty values), then the attributes remain (as is proper). This bug is consistently reproducible, and consistently fixable (when removing the empty attribute value).

    I'm using the metavms-metadata_sdk-4.2.0.32836-universal.zip, and I just tested on a local installation of nx meta 5.0.0.33871 R4. I have the plugin dll copied into the plugins folder.

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David

    Thank you for your feedback.

    after toggling the object time filter between, e.g. Any Time and Last Day, as shown in the previous video. 

    I managed to reproduce that. Continue my investigation.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Is there any update on this issue? I'm more concerned with old best shot images being applied to incorrect tracks, but wonder if these two issues are related.

    Also, regarding track confusion reported by the developer, I'd like to reiterate that it was not track confusion displayed in the logs, but was expected behaviour. The UPDATE entries are for an eternal track that changes attributes each frame, and the INSERT entries are for ephemeral tracks that have non-changing attribute values. The latter tracks are easily identified the first 8 characters of the track ID. Hopefully this makes sense; if not, there's an image here that might describes it better: https://file-store.surge.sh/attributes-insert-update.png 

    Cheers

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hello David,

    The issue has been fixed. The fix should be included to the next Nx Meta public version.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Hi Andrey,

    Are you referring to the mismatched best-shot images? Or just the disappearing attributes?

    0
    Comment actions Permalink
  • Avatar
    Andrey Terentyev

    Hi David,

    I mean the issue "Missing attributes in object list".

    If you still have any difficulties with mismatched best-shot images, let's create another thread.

    0
    Comment actions Permalink
  • Avatar
    David Brown

    Ok, thanks very much Andrey.

    0
    Comment actions Permalink

Please sign in to leave a comment.