4. Implementation Best Practices

Capturing Objects from Video

The general goal of capturing objects on video is to help the user augment their live video views with objects or to search specific moments on video using text search or region of interest. To do that, a Plugin can label specific frames with specific metadata consisting of a bounding box and a set of attributes defined by the Plugin. The list of Plugins is arbitrary, not predefined by the platform.

Best Practices for Capturing Objects

Below are a few things you can do to make sure you effectively detect and relay detected object metadata to Nx Witness via the Metadata (Analytics) SDK.

Detect Bounding Boxes (if possible)

• Bounding boxes define the location of detected objects and enable Smart Search in Nx Witness for operators - allowing them to search specific regions of the video over time for target objects.

Track the Object between Sequential Frames

• Tracking the object between frames is critical in order for Nx Witness to treat the detection event as a single sequence. Not doing this will result in hundreds of duplicate results.

Extract as many Attributes of an Object as possible.

• Attributes are stored individually in Nx Witness's dedicated object database and allow operators to search the archive more effectively.
• Attributes may include color, clothing type, name, license plate number, age, gender, or any other potentially unique characteristic of a detected object.

Send object coordinates ASAP, send attributes when possible

• It is important to send object coordinates as soon as possible to let users see them as-real-time-as-possible, and enhance the performance of an integrated video analytics solution.
• Attributes can be sent separately with a slight delay (or even if the object is not on the video anymore) as they are mainly expected to be used in search/review.

Label Video with meaningful metadata

• Feel free to attach not only object data but additional externally generated data that might be useful to an operator. Some 3rd party solutions may not be object-focused, but scene-focused. For example, a Point of Sale solution can send metadata associated with the whole frame and include the transaction data – $ amount, item sold, time, cash till amount, etc.

Creating Custom Events / Behavioral Analytics

While captured metadata Objects are useful for search, Events can be even more meaningful as they can be triggered for specific behaviors the analytic / computer vision application has been engineered to identify.

The Metadata (Analyics SDK) has been engineered, as a result, to enable the creation of Custom Events for System Operators to use in the Rules Engine of Nx Witness – a graphical user interface in Nx Witness Desktop that uses "If/Then" logic to automate System Actions – e.g. Notification, Layout-as-an-Action, send HTTP Request to a 3rd-party Access Control system to open a door, etc.

Best Practices for Creating Custom Events

Do not send events every frame

• If an event is triggered every frame, the system may be overloaded and an event triggered every frame is probably not going to be very useful/helpful to the operator.

Use prolonged Events when possible

• Longer events (activated when the behavior is detected, deactivated when the behavior ends) create useful, meaningful events that can be used to create recordings or bookmarks – typically it lasts 5 seconds or more.

Send accurate timestamps with Events

• Make sure the Events generated by a 3rd-party video analytic / computer vision application have accurate timestamps so that the detected event is reflected accurately on Nx Witness Desktop's timeline. Some events may take some time for the analytics engine to detect. That's fine – you can always send the event when detected (even if in the past), and it will be helpful to operators as long as the timestamp is correct.

Distinguish Event types

• Different Events are meaningful in different situations. For example, a person detected at the bank vault is an important event, while a person detected in the mall is not important at all. Distinguishing different event types lets the user decide what is important for him and what needs to be ignored.

Use Caption and Description fields effectively

• Every event should include a Caption and Description to be useful to operators.
• Captions generally include the most useful information as they are shown in the Notifications Panel when Events are received by Nx Witness from a Plugin. Captions should be concise and precise.
• Descriptions provide additional data for the Operators to consume and are shown in pop-up previews (when the Operator hovers their mouse over a Notification). Descriptions should be robust and can contain a lot of useful information.
• Both the Caption and Description of an Event can be used to search for / filter events.

Document all Events

• As Events are unique and may not be interpreted easily by new Operators take the time to document what each event means and the format you use for Captions and Descriptions.

Utilizing Plugin Settings

Plugins, when implemented, should be easily configurable during setup and as instantly usable as possible. In short — the simpler a configuration is for Nx Witness Desktop operators, the less setup and ongoing support issues will exist as operators install and begin to use a Plugin.

The Metadata SDK includes two flexible ways to make an application's configuration settings available to Operators in Nx Witness Desktop:

1. Inside Nx Witness Desktop — Developers can describe a list of parameters the operator should be able to modify. This will generate a form automatically for Operators in Nx Witness Desktop.
   
   
2. In a Web-Based Interface in Nx Witness Desktop's embedded browser — Nx Witness Desktop has a built-in browser that can be utilized to create a configuration interface for their Plugin. This is more flexible as it allows developers to customize how their configuration interface will look but is also a more complicated approach as it will need to be tested for compatibility and usability.

Support on-the-fly configuration

• This allows Operators to modify configurations in real-time without the need to restart any part of the combined solution during configuration/modification.

Explicitly state when a restart is required

• Sometimes it may be necessary to restart a service in order to apply a new configuration. It happens. By stating a restart is required you can enhance the user experience of Operators and avoid unnecessary support inquiries.

Avoid the need for configuration files or separate applications

• Try to avoid any prerequisite separate applications and/or the need to manually add config files during Plugin installation. Most users will fall down during these steps and this will result in additional support requests / unhappy customers.

Support camera/scene specific settings

• Analytics / Computer Vision applications work differently on different camera views.  If your analytics application needs to be configured for different scenes, make those configuration settings available in the camera settings (DeviceAgent). If a setting is universal (not specific to cameras), keep it in the Plugin settings (Engine).

Make it work out-of-the-box

• Provide default values for all settings for a Plugin so that it begins working immediately (without the need for the Operator input), if possible for most common deployment scenarios.

Make defaults flexible

• Although it's great to include default values of camera-related settings (DeviceAgent), so a solution works out-of-the-box, it's also a good idea to allow Operators to change these defaults using System-wide Plugin settings (Engine), with individual settings for specific cameras/streams.

Validate settings

• Validate your Plugin settings to avoid basic operator input errors and to provide feedback during configuration.
• Not validating introduces the opportunity to cause a Plugin crash, which results in the entire Nx Witness Server application crashing as well (Plugins run within the same OS process) – and, well, nobody wants that.
• Send validation information as Plugin events.

Plugin Diagnostic Events — reporting errors and diagnostic information

Nx Witness's Metadata SDK also includes the ability to send Plugin health events to Nx Witness to alert Operators whenever a Plugin is not performing as expected, e.g. the video analytics engine backend is offline.

Any health events sent from a Plugin will be displayed as Notifications in the Notifications Panel.

Plugin Diagnostic Events - Best Practices

Send Events any time an issue occurs

• Help keep operators informed about performance issues and allow them to act to resolve issues as early as possible.

Make the Event description actionable

• Inform the Operator what needs to be done to resolve the issue (if anything). For example, mention a specific setting that may cause the issue and suggest how to address the issue.

Report Service Health State Changes

• In many instances, the integrated 3rd-party video analytics/computer vision application may be dependent on a remote service. If that service becomes unavailable, let the Operator know that the resource has gone offline and update them again when it is back online.

Implement Debug / Diagnostic Mode

• It is a good idea to use Plugin detailed events for debugging / diagnostic purposes. If you decide this is something you'd like to do, we suggest you add a Plugin setting that will trigger the diagnostic mode. Make sure to send as much information in the Event Description field as possible in that mode.