Is the Nx Cloud up? Visit our Status Page for the current health and performance of the Nx Cloud.

Status Page

Issue with timestamp in rtsp stream (in combination with plugin)

Answered
Permanently deleted user

Some context:
We are working on a framework that allows to scale processing over multiple nodes (using Ray). We have several processing pipelines available using this framework (car detection, car tracking, pedestrian monitoring, ...) that we want to connect on Nx VMS, and collect the detection/tracking information in Nx. Important to note, since the processing could require multiple nodes, it will not run on the same system as the Nx server.

To do this, I'm using the following strategy:

  • Use an rtsp stream (e.g. rtsp://admin:PASSWORD@192.168.0.184:7011/UUID?onvif_replay=1), since this allows to send video data over the network more efficiently compared to reading the frames in the plugin and send them over the network one by one.
  • To allow reading the timestamp from the stream, we use a wrapper around an adapted version of PyAV (https://github.com/PyAV-Org/PyAV/pull/672).
  • We send meta-information over a message-bus (currently zmq), with for each detection the timestamp of the frame we have read
  • the plugin code listens on the same message-bus and calls "pushMetaDataPacket"

I have some issues though:

  • Although the timestamp comes from the stream, there is still a small delay on the metadata in comparison with the overlay shown in the Nx client (more or less 1.5 seconds). For this experiment I run a face detection model on a network camera on my desk (easy for experiments)
  • For a larger experiment, I use a virtual camera using a videofile. For this I adapted the url to also contain the timestamp (rtsp://admin:PASSWORD@192.168.0.184:7011/UUID?onvif_replay=1&pos=2020-06-30T14:53:00:00.000). The timestamp from the stream is always the same for all frames though, which is the timestamp corresponding to "2020-06-30T14:52:59:00.000" (so more or less a second earlier, I truncated to second precision)

So some questions:

  • Any idea where the 1.5 second synchronization mismatch could come from? Since the obtained timestamp changes for each frame, I suppose this timestamp is interpolated between 2 keyframes?
  • Could the timestamp not be used for virtual cameras?

Best regards,
Floris

0

Comments

4 comments

Date Votes
  • Permanently deleted user

    If the metadata arrives after the frame was shown, the Desktop client still shows the bounding box, but in archive mode, this delay should go away.

    Can you please check:

    1. Do you see the same delay when you watch archive video?

    2. What latency does your pipeline introduce?

     

     

    0
  • Norman
    • Network Optix team

    Hi @...,

    Just checking in to make sure you received our response as it has been a while since we have heard from you. Please let us know if you still need help by responding to this message.

    0
  • Permanently deleted user

    Hi,

    sorry for the late reply, I have been really swamped with other projects the past weeks (and still). I did not find the time to do a deeper analysis, but tried to get the information to answer your questions.

    1. In Nx meta client, I always see some delay between moving in front of the camera, and the live feed in the Nx meta client. Since my issue is with a mismatch between the timestamp I get from the rtsp stream, and the timestamp the meta-information is shown on, I also did the processing on archive footage (request an rtsp stream from a few minutes ago instead of the live feed). There I still see a delay between the bounding box shown in the client, and the video footage. So, when I use a face detector and I move my head from left to right, the bounding box is always a little behind (more or less 1.5 seconds). Does that answer your question?

    2. Currently I use a face detector for my experiments. On GPU in runs at around 10fps. The overhead of retrieving the frame and sending the meta-information over zmq seems negligible (a few milliseconds). 

    Best regards

    0
  • Andrey Terentyev
    • Network Optix team
    Edited

    Hello Floris,

    Which version of Nx are you using?
    > Any idea where the 1.5 second synchronization mismatch could come from?

    > Although the timestamp comes from the stream, there is still a small delay on the metadata in comparison with the overlay shown in the Nx client (more or less 1.5 seconds). For this experiment I run a face detection model on a network camera on my desk (easy for experiments)

    This delay most definitely is introduced by your plugin/model. I'll try to explain possible root cause.

    In version 4.1 there is a special dynamic buffer with the max delay of 2 seconds. The Server waits a metadata from a plugin for this period of time in order to write the metadata just next to the frame it belongs to. Nevertheless, the timestamp should not change in that case neither, excepting the case when a plugin changes the timestamp explicitly for some reason.

    In other words, when a plugin gets a frame from the Server, the timestamp is already set by the Server. A plugin should not change frame's timestamp.

    In your scenario, you have to make sure your system reads by PyAV the timestamp of the same frame it is actually processing.

    Let's consider an example.

    At the moment of time T0 the Server sends a frame with the timestamp TS0 to RTSP stream.
    At the same moment the Server pushes the same frame to the plugin for processing.
    It does not matter if your plugin does not actually process the frame, the frame is pushed with the timestamp TS0 set.
    That means the pushUncompressedVideoFrame or pushCompressedVideoFrame returns immediately making the Server wait for some time before writing the frame to the stream and sending to GUI.

    Your system receives the frame at the moment of time T1=T0+delay1 = TS0+delay1
    Your system processes the frame and sends the metadata to the plugin at the moment of time T2 = T1 + delay2.
    This metadata should contain the timestamp value of TS0.
    Your plugin receives the metadata at the moment of time T3 = T2 + delay3 = TS0 + delay1 + delay2 + delay3.
    Your plugin should pass to the Server the metadata with the timestamp value of TS0. If it's done within max 2 sec the metadata is written next to the frame it belongs to and both are sent to GUI.

    > Currently I use a face detector for my experiments. On GPU in runs at around 10fps. The overhead of retrieving the frame and sending the meta-information over zmq seems negligible (a few milliseconds).
    Taking this into account, I'd suppose there is a kind of bug inside your plugin code (or inside the external system code) setting the wrong timestamp.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post