This document explains how the Server interacts with integrations during video processing, recording, and metadata generation.

Video frame processing

When a video device initializes, the Server creates one Stream Reader for that device to read encoded video frames.

The Server handles these frames based on your configuration:

• Motion detection: If enabled, the Server always decodes the video frames.

• Recording: If enabled, the Server writes the encoded video stream directly to the video archive without transcoding it.

Integration initialization

When you enable a integration for a device, the Server automatically creates the following components for that device:

• One DeviceAgent class instance.

• One dedicated Muxer instance assigned to that DeviceAgent.

Requesting video frames

Integrations do not receive video frames by default. If your integration requires frames, you must update the capabilities property in your Engine manifest.

For example, to request uncompressed YUV420 frames:

"capabilities": "needUncompressedVideoFrames_yuv420"

When enabled, the Server sends frames to the DeviceAgent by calling either pushUncompressedVideoFrame() or pushCompressedVideoFrame().

Metadata generation and streaming

After processing a frame, the DeviceAgent can generate object and event metadata. It passes this data back to the Server as pointers to ObjectMetadataPacket and EventMetadataPacket instances.

The Server then processes this metadata through two parallel tracks:

• Analytics DB: Saves the metadata directly to the database to make it Available for Search.

• Muxer: Sends the metadata to the Muxer to Combine with Video, which simultaneously:

  • Streams to the Desktop Client for the live stream.

  • Saves to the Archive as an .mkv file.

Live streaming requirements

To display metadata in the Desktop client live stream, the end-to-end delay must not exceed 2 seconds.

Archive storage and buffer limits

When the Muxer accepts a metadata packet (any pointer to IMetadataPacket), it buffers the packet for up to 5 seconds to allow for reordering. This buffer limit minimizes system memory consumption.

The Muxer then writes the metadata to a separate track within the .mkv video file while the Server indexes the timestamps in the database for searching.

Handling processing delays

To ensure your metadata syncs correctly, keep processing delays under 5 seconds. Delays affect the system in the following ways:

• Delays of 5 seconds or less: Metadata appears properly aligned with the video during GUI playback and remains fully searchable.

• Delays over 5 seconds: The Server writes the metadata to a different .mkv chunk. While the metadata track stores a valid timestamp and remains fully searchable via the database, the metadata will not display during video playback in the GUI.