Metadata SDK : synchronize frames from all cameras

Answered

Mat L

June 17, 2020 16:14 Edited

Hi,

I'm currently developing a Metadata SDK plugin that will send frames to a deep neural network.
The input data processed by the neural network is a batch of synchronized frames from multiple cameras.
Synchronization means the time lag between frames of the same batch must not exceed 100 ms.

As device agents seem to run independently from each other, I'm not sure it's possible to gather synchronized frames from every cameras inside the plugin.
That's why I considered sending frames and timestamps to another process that will take care of the synchronization task.

Before developing it, I just want to make sure I'm not overlooking any alternative.
Is it possible to perform this synchronization task with the Metadata SDK ?

0

• 

  Permanently deleted user

  June 18, 2020 21:00

  Hi Mat, I think the solution is to track timestamp with the frame within your processing pipeline. 

  Frames from different cameras have different timestamps. Also, each camera and network environment introduce random fluctuation, delays, and frame drops.

  So perfect synchronization is impossible, and waiting for frames to sync them is not an option either.

   

  The only robust solution is to send to pipeline whatever is available, and track the timestamps along the way, so that once you have bounding box - you have associated timestamp and same with detected attributes.

  1
• 

  Mat L

  June 19, 2020 07:50 Edited

  Thanks for your answer!

  I'm fine with imperfect synchronization, as long as it doesn't exceed 100 ms.
  I agree with the solution you're suggesting.
  
  Can you confirm the exact meaning of videoFrame->timestampUs()?
  If "trust camera timestamp" is disabled, is it the timestamp when the server receives the frame? If "trust camera timestamp" is enabled, is it the timestamp provided by the camera (in the RTSP headers, I guess)?

  0
• 

  Andrey Terentyev

  • Network Optix team

  June 19, 2020 14:22 Edited

  Hello,

  > If "trust camera timestamp" is disabled, is it the timestamp when the server receives the frame? If "trust camera timestamp" is enabled, is it the timestamp provided by the camera (in the RTSP headers, I guess)?

  Yes, that's correct.

  > As device agents seem to run independently from each other, I'm not sure it's possible to gather synchronized frames from every cameras inside the plugin.

  Although each device agent runs independently, all of them have one common Engine class instance as a parent.

  You could synchronize frames from different device agents in the Engine thread.

  I'll try to sketch up a possible solution.
  So, let's assume you have a plugin (P) on your server (S) with three cameras C1,C2,C3 with device agents DA1, DA2, DA3.

  You can modify device agent constructor in a way it accepts a pointer to Engine.

  DeviceAgent::DeviceAgent(Engine* engine, const nx::sdk::IDeviceInfo* deviceInfo)

  and pass the pointer in Engine::doObtainDeviceAgent

  void Engine::doObtainDeviceAgent(Result<IDeviceAgent*>* outResult, const IDeviceInfo* deviceInfo)
  {
  *outResult = new DeviceAgent(this, deviceInfo);
  }

  See stub_analytics_plugin for details.

  Possible device agent should have the m_engine member for storing a pointer.

  In Engine class you could add a private member of queue type. And the addItem  method for adding a countable pointer to a frame to that queue. Something like

  addItem(const IUncompressedVideoFrame* videoFrame)

  Mind to have a mutex defined in order to sync addition to the queue, since the addItem method will be called from different independent threads.

  Each device agent woulld call

  m_engine->addItem(videoFrame)

  One thing remains is to create a thread in Engine constructor, which would process the queue, find frames with timestamps fitting a time slot of 100ms, put them in a batch and send to external application.

  1
• 

  Mat L

  June 19, 2020 15:27

  Thanks, Andrey, for outlining this solution.
  It sounds like a great way of synchronizing the frames "inside" the plugin. I'll try to implement it.

  0

Please sign in to leave a comment.