9. Important Technical Considerations w/ Nx Witness

Best Practices for Processing Video

Track objects using at least 10 frames per second

• Processing video at less than 10 fps makes tracking unreliable.

Optimize attributes extraction

• Most frames captured by real-world IP cameras may be blurred and may not be suitable for feature extraction.
• To optimize resource usage and reduce false positives, run attributes extraction only on pre-qualified, high-quality frames if possible.

Keep frame latency low

• Frame latency will impact how the user sees metadata on live video. The lower the fame latency, the better the object tracking and associated metadata will be reflected on Nx Witness Desktop.
• To address this issue, Nx Witness Desktop has a buffer of 500 ms for live video before decoding for the Operator. If you aim for a target latency of 300 ms, you will prevent Operator visual issues.
• If for any reason you are not able to maintain low latency, the captured metadata will still be usable and reviewable on archived video, but will not work well on live video.

Drop frames, do not build the queue

• This is CRUCIAL. If Nx Witness Server does not have enough resources to process frames in real-time, building the queue will result in increased RAM usage and can result in a Server application crash.
• The best way to address this issue is to keep the queue size limited and drop the oldest frames in the pipeline.
• It's a good policy in general to drop all frames which are older than 300 ms.
• NOTE – Nx Witness Server does not drop frames. It will send all frames to the Plugin as requested and will expect the Plugin to make the decision about which frames to drop.

Real-world Camera Considerations

IP cameras come in various shapes and sizes, but they all pretty much work in the same way. There are a few basic realities when dealing with IP cameras, which need to be handled by the Plugin graciously.

Camera availability is not persistent

IP Cameras can be added or removed from the System – either on purpose or because of device failure. This means resources may become unavailable through no human intervention.

Network infrastructure is inconsistent

In poor network environments, dropped/lost frames and big latency can be an issue.

Stream quality / FPS may change

Users may reconfigure cameras on the fly, which can affect the resolution, frame rate, and bitrate of available streams.

Every IP camera is unique

Manufacturers do a lot of tricks to get their cameras to perform – including using different codecs, resolutions, GOP-size, digital shutter intervals, etc.

When developing an Nx Witness Plugin, make sure it's as camera-proof as possible: plan methods of correctly handling the conditions listed above.  If something specific cannot be supported by the Plugin – send relevant information to the user using a Plugin Event, but aim to create a Plugin that never crashes.

Implementing Cross-Platform Support

If your Video Analytics solution can work on more than one platform, creating a cross-platform Plugin will increase the applicability for your solution for a wider range of customers.

Metadata (Analytics) SDK is cross-platform, which means that you can create integration once and compile it for any platforms you support.

The recommended priorities (based on market share/feasibility):

• Windows x64 (most of the hardware servers out there)
• Ubuntu x64 (popular in large systems)
• ARM-64 (Nvidia Jetson and other smart NVRs)
• ARM-32 (Raspberry Pi and low-cost edge NVRs)

Make sure that the Plugin package has all required dependencies and data files for every platform, or a simple script to install them.