HTTP Generic Event timestamp synchronization
Answered
Hi,
We read a rtsp video stream* in our third party application. Then, we detect objects in the stream frame by frame. Finally, we create and HTTP generic event using a timestamp in milliseconds. However, we are unable to synchronize the event created in Nx MetaVMS and the frames we have in our third party application. There is a delay of some seconds (1-4 sec) between the frame we use for creating the event and the frame displayed in the Nx MetaVMS events screen.
We have tried several strategies for setting this timestamp in our application:
1. To use the local timestamp we have when we read the frame.
2. To use the presentation timestamp information provided by the rtsp stream.
3. To use the decoding timestamp information provided by the rtsp stream.
None of them have worked and we have also tried the option "Trust camera timestamp". So how do you get the frame when you received an HTTP event? which timestamp do you use?
*This stream is generated by your utility testcamera, using always the same test video in order to ensure the reproducibility of the experiment.
-
Hi Eduardo,
Preview generated for Generic events are taken from the closest key-frame, they are not using exact timestamps. This is done to optimize resources on the server.
The best way to solve the problem is to integrate through the SDK, in this case, the previews will be precise.
-
Great, so we have modified our pipeline to do the following:
1. Open the stream using the URL:
rtsp://user:password@192.168.2.105:7001/e3e9a385-7fe0-3ba5-5482-a86cde7faf48?onvif_replay=1
2. Grab video frames in our third party application using live555. This library provides synchronised presentation times using the RTCP sender reports [1][2]. We use this timestamp as the absolute timestamps.
3. We send objects with the corresponding timestamp information to a plugin running on nx metaVMS server, that expose a REST API.
4. This plugin creates the corresponding nx object using the absolute timestamp we sent before.
The behaviour is better but we still find an important delay between what we see in our third party application (using live555) and the thumbnails in the nx metaVMS.
[1] http://lists.live555.com/pipermail/live-devel/2008-July/009224.html
[2] https://live-devel.live.narkive.com/WgLSYFbC/synchronize-the-rtcp-time-using-rtsp-range
-
Hello Eduardo,
Though, your described pipeline is possible, the recommended one is presented hereafter in order to minimize delays.
1. Create analytics plugin by means of Metadata SDK. The plugin runs on the same machine the Server does, in the same process the Server does. Once a plugin is enabled on a camera, the Server passes all the video frames from this camera to the plugin. This happens in the same process, not network is involved. This reduces delays a lot.
Your plugin sends either all the frames or just several of them to "third party application" your remote machine. It could significantly reduce the delays, too:
a) You can detect objects not in every frame, but just in selected. If you don't want to perform complete frame processing, let's call this step pre-processing.
b) once objects are pre-processed, this particular frames only could be sent for further processing to "third party application" over the network.
2. "Third party application" processes the frames and sends back to the plugin object metadata.
3. On receive, the plugin passes object metadata to the Server.
Here is the scheme.
Nx Witness Server -1. passes frames-> analytics plugin ---2. sends frames over the network---> "third party application"
"third party application" --3. sends metadata over the network--> analytics plugin -4. passes metadata-> Nx Witness Server
Here is the similar case.
SCENARIO
-
Thank you very much for your prompt response.
I am not concerned about the delay, but rather about the synchronisation. I can afford a small delay in the visualisation, but unsynchronization of object detections and frames does not look good. Therefore, I want to further develop the scenario I proposed. Moreover, I want to avoid (as far as possible) the scenario where I send the frames from the plugin through a socket (or anything like that) to my 3rd party application, since it seems to me like reinventing the wheel... we are already using RTSP, it should be possible to synchronise some metadata on top of the stream I think.
So, I wonder... how do you grab the timestamp within the server for the following scenarios?:
1. Trust camera timestamps disabled: I guess that you get it from the system clock.
2. Trust camera timestamps enabled: here we are using the method setTimestampUs within the plugin, using the timestamp we grabbed from the stream in our 3rd party application with live555. Is it compatible with your implementation?
-
1. Trust camera timestamps disabled: Server ignores timestamps from the camera and assigns server time to every frame.
2. Trust camera timestamps enabled: Server keeps timestamps from the camera as long as the difference between camera time an server time is less than 10 seconds.
Regarding the synchronization issue: Can you please create a short video demonstrating the problem? (if you are running Nx Client on Windows, you can use the screen recording feature to do it)
To debug the problem further we need to understand, which specific component introduces discrepancy.
Can you try and track a single frame in the pipeline from beginning to the end saving timestamp at every step?
Also, you can request a precise image from the server for any given timestamp, compare it to saved frame a and see if server returns the image you saved in the pipeline.
-
I have performed two experiments in order to debug further the issue.
Experiment 1
For this experiment, I stream with your application testcamera a low framerate video, where I overlaid the frame number at the bottom. I blurred this video for privacy reasons.
I have tracked some frames, comparing the thumbnails of nx metaVMS at the left, and the images of my pipeline at the right. You can see a difference of 2-4 frames between the server and my pipeline images, which means around 600 ms of delay.
Experiment 2
For this experiment, I use a IP RTSP camera. I have recorded a physical chronometer and I have compared images of the server and my pipeline, measuring a delay of 600-700 ms between both. In this experiment, the images presented at the nx metaVMS are late (compared to thus of my pipeline).
Moreover, I have noticed no difference using or not the option "Trust camera timestamp". I think it makes sense, since I capture the stream generated by nx metaVMS from my pipeline.
-
Solution from a support ticket.
For the live stream the Nx Desktop displays object metadata and thumbnails on the right-hand panel as it receives those, i.e. without regarding or syncing to what frames are displayed on the scene.
So, there might be a situation when you see object thumbnail and metadata on the right-hand panel but don't see a corresponding bounding box on the live picture on the scene, because the metadata is too late to be displayed on the live frame, since this frame is already outdated and can't be displayed live anymore.
There is an intentionally introduced delay of max 2 seconds before the frame is displayed, exactly in order to address this latency between the metadata and the frame.
This delay is dynamically calculated depending on the actual metadata arrival statistics.
So, you might see a delay of max 2 seconds between physical picture and it's representation in the live stream on the scene in Nx Desktop.LIVEDELAY
Please sign in to leave a comment.
Comments
11 comments