Articles in this section

See more

Troubleshooting Analytics Issues for Cameras and Plugins

Avatar
Nx Support
  • Updated
Follow

This article describes how to enable and use analytics logging to assist in troubleshooting video analytics issues. 

The values used in the example commands and outputs below are based on the cameras and plugins used in our test environment and will need to be replaced with the appropriate values from your own environment. The portion of the command which needs to be replaced with your information are bolded.

Prerequisites:

Enable Analytics Logging

  1. Create the nx_ini directory for both Nx Server and Nx Desktop.

    To determine the expected file path for nx_ini, please refer to the “How it works” section of the Configuring via .ini files – IniConfig support article.

    The default location of the nx_ini directory is expected to be:
    Nx Server
    • Windows:
       
      C:\Windows\System32\config\systemprofile\AppData\Local\nx_ini
    • Linux:
       
      /home/networkoptix/.config/nx_ini

    Nx Desktop
    • Windows:
       
      C:\Users\%USERNAME%\AppData\Local\nx_ini
    • Linux:
       
      /home/$USER/.config/nx_ini
    •  
  2. Create the analytics_logging.ini file in the directory.

    It is better to create an empty analytics_logging.ini file, as both Nx Server and Nx Desktop will fill their respective files with the default values.

  3. Change the settings in the analytics_logging.ini file.

    Before opening the analytics_logging.ini file and changing its settings, create the analytics_log_dir directory.

    analyticsLogPath=analytics_log_dir
    logObjectMetadataDetails=true

    Examples
    • Windows:
       
      analyticsLogPath="C:\\analytics.logs"
    • Linux:
       
      analyticsLogPath="/tmp/analytics.logs"

    The analytics_log_dir path is either relative to the nx_ini directory, or an absolute path. The logObjectMetadataDetails setting forces object attributes (including trackId) to be written to the analytics log files.

Live Objects Do Not Appear in the Notification Panel

Please follow the steps below in Nx Desktop to troubleshoot the issue:

  1. Right-click the desired camera and open Camera Settings. Make sure recording is turned on and the plugin is enabled.
  2. In the Notification Panel, make sure the search field is empty and no filter is selected.
  3. Check to see if Nx Desktop receives the object metadata:
    a. On the client machine, go to Nx Desktop’s analytics_log_dir directory. 
    b. Search the rtp_parser_AnalyticsPanel_*.log files for the typeId (e.g. nx.stub.car) of the object that does not appear:
    grep -B1 nx.stub.car rtp_parser_AnalyticsPanel*.log
    c. If object data is in the log file but is not displayed, it is most likely a bug in Nx Desktop.
  4. If object data is in the log file, search the log files downwards through Nx Server’s metadata pipeline as described in the Analytics Logs: Monitoring Metadata Flow.
  5. If object data is NOT in the log file, make sure the Server receives object data from your plugin.
    a. On the Server machine, go to Nx Server’s analytics_log_dir directory.
    b. Search the outgoing_metadata_device_*.log files for the typeId (e.g. nx.stub.car) of the object that does not appear:
    grep -B1 nx.stub.car outgoing_metadata_device_*.log
    c. If object data is NOT in the log file, your plugin does not pass data to Nx Server. Investigate your plugin code starting with the pullMetadataPackets() method.

Live Objects Being Searched Do Not Appear in the Notification Panel

Please follow the steps below in Nx Desktop to troubleshoot the issue:

  1. Right-click the desired camera and open Camera Settings. Make sure recording is turned on and the plugin is enabled.
  2. Using the Server Monitor in Nx Desktop, make sure the storage drive that contains the analytics database is not overloaded.
  3. Check to see if Nx Server writes object metadata to the analytics database.
    a. On the Server machine, go to Nx Server’s analytics_log_dir directory.
    b. Search the analytics_event_receptor*.log files for the typeId (nx.stub.car) of the object that does not appear:
    grep -B1 nx.stub.car analytics_event_receptor*.log
  4. If object data is NOT in the log file, it is likely your plugin does not pass data to Nx Server. Investigate your plugin code starting with the pullMetadataPackets() method.

Object Preview Image Is Wrong

image1.png

Sometimes the best shot image is from

  • a different camera than the one selected, or
  • the correct camera but at the wrong timestamp (days/weeks from the actual timestamp of the detected object).

To show the preview image, Nx Desktop invokes the Server API passing trackId.

The trackId is taken from the

  • best shot packet of a track (there can be only one best shot packet per track), or
  • the first metadata packet of a track (only if there is no best shot packet for the track).

The most common root cause of an incorrect object preview image is track id duplication. The fastest way to prove it is to search through the analytics database for duplicated records containing the same track id.

The analytics DB is stored in the object_detection.sqlite file in the video archive folder.

Use this command to get a table with duplicated track ids:

sqlite3 -readonly object_detection.sqlite "select quote(guid), count(1)
from track
group by guid
having count(1) > 1"

The second most common root cause an incorrect track id passed either in the metadata packet or in the best shot packet. To figure it out, we need to analyze the analytics logs:

  1. Identify the "wrong" object by searching for it using timestamps, attribute values, object types, etc.
    (See Live Objects Do Not Appear in the Notification Panel.)
    a. Find the Camera ID. For example, the testcamera in the screenshot above is e3e9a385-7fe0-3ba5-5482-a86cde7faf48.
    b. Search for trackId entries in the file: 
    grep trackId rtp_parser_AnalyticsPanel*e3e9a385-7fe0-3ba5-5482-a86cde7faf48*
    c. View the partial output:
    metadataTimestampMs 1606420996442,
    currentTimeMs 1606421067433, diffFromPrevMs 3669, diffFromCurrentTimeMs -70991;
    bestShot:     x 0.423177, y 0.622222, width 0.565365, height 0.351389,
    trackId {0a000000-0000-0000-0000-000000000000}
  2. Copy the trackId from the found object (bolded in step 1.c): {0a000000-0000-0000-0000-000000000000}
  3. Find duplicates with the same trackId in the log files with a different CameraID. 
    a. Use the /ec2/analyticsLookupObjectTracks Server API function.

    It should look similar to this:
    https://localhost:7001/ec2/analyticsLookupObjectTracks?objectTrackId=%7B0a000000-0000-0000-0000-000000000000%7D
    b. View the output:image2.png

As seen in the image above, the "Hanwha LPR" camera has an object track with the same id, the best shot of which has been taken for the preview image.

You can automate steps 1-3 with a script by iterating through the list of unique trackIds found in step 1 and comparing the deviceId in the API output against the Camera ID of the camera in question.

Objects Appear with a Misplaced Bounding Box

The most likely cause of the issue is an incorrect bounding box calculation. For debugging, see the Live Objects Do Not Appear in the Notification Panel section and complete the steps below to compare bounding box coordinates.

  1. Look for an object of the nx.stub.car type that contains the string “Ford” in its attribute values: 
    grep -B1 nx.stub.car widget_analytics_controller_device*.log | grep -B1 Ford
    a. View the output: 
    metadataTimestampMs 1606244933579, currentTimeMs 1606244937617, diffFromPrevMs 166, diffFromCurrentTimeMs -4037; objects: 1:

      x 0.578564, y 0.313223, width 0.392294, height 0.137392, trackId {ab276a67-771f-4136-aa57-17fd9f5600e2}, typeId nx.stub.car, attributes {Brand: "Ford", Model: "Focus", Color: "Black", SomeValue.sys.hidden: "a hidden value", Distance: "0,657909", Random: "14,000000 sqft"}
    b. Take note of the bounding box coordinates from the output:
    • x 0.578564
    • y 0.313223
    • width 0.392294
    • height 0.137392
  2. Search for the 1606244933579 timestamp value: 
    grep -A1 1606244933579 outgoing_metadata_device_*r
    a. View the output: 
    outgoing_metadata_device_f92adeff-3ac0-fb7c-d793-06b5a3475b52_engine_687611a2-fd30-94e7-7f4c-8705642b0bcc.log:metadataTimestampMs 1606244933579, currentTimeMs 1606244936135, diffFromPrevMs 166, diffFromCurrentTimeMs -2555; objects: 1:

    outgoing_metadata_device_f92adeff-3ac0-fb7c-d793-06b5a3475b52_engine_687611a2-fd30-94e7-7f4c-8705642b0bcc.log- x 0.578564, y 0.313223, width 0.392294, height 0.137392, trackId {ab276a67-771f-4136-aa57-17fd9f5600e2}, typeId nx.stub.car, attributes {Brand: "Ford", Model: "Focus", Color: "Black", SomeValue.sys.hidden: "a hidden value", Distance: "0,657909", Random: "14,000000 sqft"}
    b. Take note of the bounding box coordinates from the output:
    • x 0.578564
    • y 0.313223
    • width 0.392294
    • height 0.137392

  3. If the bounding box coordinates are the same, check your plugin code for calculating bounding box coordinates, as there may be an issue there.

Objects Are Detected but with Misplaced Bounding Box in Preview

image3.png

See the Object Preview Image Is Wrong section.

Object Attributes Are Not Displayed

The most likely root cause is that attributes are not sent to the Nx Desktop. See Live Objects Do Not Appear in the Notification Panel.

Make sure at step 5 there are attributes values in the search result.

Object Bounding Box Is Not Displayed

The most likely root cause is that the delay between the moment the Nx Desktop receives the object metadata packet and the moment Nx Desktop receives the video frame which the metadata corresponds to is more than 2 seconds.

For debugging see the Live Objects Do Not Appear in the Notification Panel section. 

  1. To diagnose the issue of a missing bounding box for an object, look for the object using its attributes. In our example the attributes will be Brand: "Lada", Model: "Granta":
    grep -A1 Granta rtp_parser_AnalyticsPanel*
  2. From the output, determine the following:
        • The currentTimeMs value the time metadata has arrived to the Nx Desktop analytics panel.
        • The metadataTimestampMs value.
    a. View the output: 
    metadataTimestampMs 1606242285446, currentTimeMs 1606242289687, diffFromPrevMs 166, diffFromCurrentTimeMs -2640; objects: 1:

    x 0.320696, y 0.460545, width 0.364321, height 0.343959, trackId {1f374983-0997-4565-afcd-e8dcb44740c3}, typeId nx.stub.car, attributes {Brand: "Lada", Model: "Granta", Color: "Red", SomeValue.sys.hidden: "a hidden value", Distance: "0,561202", Random: "16,000000 sqft"}
    b. Take note of the metadataTimestampMs and currentTimeMs values:
        • metadataTimestampMs1606242285446
        • currentTimeMs1606242288087
  3. Find the frame with the timestamp value equal to the metadataTimestampMs 1606242285446 and determine its currentTimeMs value: 
    grep -A1 -i "frameTimestampMs 1606242285446" rtp_parser_ClientCamera*
    a. View the output:
    frameTimestampMs 1606242285446, currentTimeMs 1606242287521, diffFromPrevMs 33, diffFromCurrentTimeMs -2074
    b. Take note of the currentTimeMs value: 1606242287521
  4. Calculate the difference between the frame’s currentTimeMs value and the metadata's currentTimeMs value: 1606242287521 - 1606242289687 = -2166
  5. The bounding box was not displayed because the value of -2166 means the delay the metadata arrived with is 2166 milliseconds compared to the arrival of the corresponding video frame (i.e. more than 2 seconds).

Related articles

Comments

0 comments

Article is closed for comments.