Plugin Error Handling

There are multiple situations when something unexpected happens, and the Plugin must make a decision on how to continue.

The recommendations we give in this article help to make Plugins robust; crashing as rarely as possible, even if some job cannot be done due to reasons internal or external to the Plugin.

The idea is that the Plugin code should be written in a way that if some anomaly is detected, it is reported in the best way possible, and the execution continues in some reasonable manner (having contained the error as tightly as possible).

For example, if some function called by Nx Witness Server cannot do its job, it must not crash but rather return an error. And if some code crashes due to some invariant violation (e.g. a pointer passed to the function can be null), the invariant must be checked and a recovery action must be taken instead of crashing — like returning an error, or omitting some part of the job.

Note: Because Plugins are dynamic libraries working as a part of the Server process, any crash inside a Plugin crashes the whole Server process. Nx Witness Server will be automatically restarted, but the conditions which led to the crash may still be in place, so this scenario may repeat in a loop.

Typical error situations that may happen in the Plugin code are listed below, with recommendations on how to handle them. Each recommendation is described in detail in a dedicated section of this chapter.

• An internal error in the Plugin code is detected (e.g. some invariant fails).
  • Fail an assertion and recover.
• A misbehavior of Nx Witness Server is detected (e.g. some function arguments supplied by the Server are incorrect).
  • Fail an assertion and recover, most likely returning an error.
• An issue with the Plugin’s backend (some software external to the Plugin) is detected.
  • If the current function can return an error appropriate to this particular situation:
    • Return an error.
  • Otherwise, including asynchronous errors:
    • Raise a Plugin Diagnostic Event and recover.
• An issue with some user-supplied data like Settings values is detected.
  • If it is the place where the data first appears in the Plugin:
    • Return an error.
  • Otherwise:
    • Fail an assertion and recover.

Note: Using the C++ exceptions is possible only inside the Plugin code — due to potentially different compilers and standard library implementations of Nx Witness Server and the Plugin, exceptions must not be thrown out of the interface methods.

The samples provided with the SDK do not use exceptions even internally; though this is not discouraged, provided that all exceptions are caught before returning from an implementation of a virtual method of an interface.

Returning errors

The most natural thing a function can do, when something goes wrong, is returning an error instead of a useful value. Most of the functions of SDK interfaces are designed to be able to return either a valid value, or an error in the form of an error code and an error message.

The code that allows you to do it is located in src/nx/sdk/result.h. There is 

• an enum nx::sdk::ErrorCode with codes for typical error conditions and a special value for all others (otherError),
• a helper class nx::sdk::Error which is simply a combination of an error code and an error message, and
• a template of a class nx::sdk::Result<Value> which represents an exclusive combination (like a C union) of an Error and a Value. This template is used for the return value of most interface methods.

Also, there is a helper function in src/nx/sdk/error, which constructs the Error object:

Error nx::sdk::error(ErrorCode errorCode, std::string errorMessage)

Because Result<Value> has implicit constructors from Value and from Error, the functions can simply return either an error or a value, using the following syntax:

Result<int> ISomething::divide(int numerator, int denominator)
{
    if (denominator == 0)
        return error(ErrorCode::invalidParams, "Incorrect number");
    const int value = numerator / denominator;
    return value;
}

Note: There is no need to log something when returning an error, unless there are some details to report which should not be included in the error message.

Nx Witness Server will log every situation where a Plugin’s function returns an error; if the Server considers it helpful to inform the VMS user/administrator about the error, it will additionally raise a Plugin Diagnostic Event (see the description of this mechanism in the dedicated section below).

Assertions

In the case that the Plugin code detects some situation which must not normally take place — and may lead to a crash or otherwise indicates an internal error in the Plugin code — it is recommended to fail an assertion. 

There are many possible assertion mechanisms, including the one in the C/C++ standard library — the macro assert(). But most of such mechanisms follow the pattern: in Release build configuration, the conditions are not checked at all; and in Debug build configuration, the conditions are checked and the process is intentionally crashed on failure.

This has the following significant issue in Release build configuration, which is the natural choice for the production use of the Plugin: the conditions are not checked, and the execution continues even if the condition would fail, which often leads to crashes.

To overcome these issues, the SDK offers its own assertion mechanism which we recommend using in Plugins to minimize crashes and improve the Nx Witness user experience: NX_KIT_ASSERT() in nx/kit/debug.h. This macro works even in Release build configuration but, in this case, it only logs the issue without intentionally crashing the process.

Also, this macro can be used as an expression returning the boolean value of the condition, enabling easy recovery instead of crashing — without the need to repeat checking the condition.

The example of a function, shown above, can be improved with an assertion like this:

Result<int> ISomething::divide(int numerator, int denominator)
{
    if (!NX_KIT_ASSERT(denominator != 0))
        return error(ErrorCode::invalidParams, "Incorrect number");

    const int value = numerator / denominator; //< Will never crash here.
    return value;
}

Plugin Diagnostic Events

When something goes wrong in the Plugin’s backend — and there is no straightforward way to return an appropriate error code and message from a Plugin’s function — the Plugin may decide to raise a Plugin Diagnostic Event. These Events are processed by the Event and Rule Engine of Nx Witness; the VMS administrator can assign Action to these Events like sending an email or notify the user (default Action).

The general idea is that the Plugin should raise a Plugin Diagnostic Event when it needs the attention of the VMS user/administrator; or, when the Plugin expects that a Rule was created, in which it assigns a selected Action to the specified type of Plugin Diagnostic Event.

In the case that a Plugin’s function can return an appropriate error code and message, there is no need to additionally raise a Plugin Diagnostic Event. In most cases, this event will be raised by Nx Witness Server in response to the returned error, to inform the user that something went wrong in the Plugin.

Also, Nx Witness Server raises Plugin Diagnostic Events when the Plugin violates its contract either by returning an invalid value, or by calling a callback function incorrectly.

If the VMS administrator did not assign some other Action for these Events, the user will see a notification about the Event in the Notification Panel and, as with all Events, the Event Log will contain a record about this Event.

Note that Plugin Diagnostic Events are not intended to represent metadata, like real-life events detected on the video or coming from some sensors ­­­— for this purpose, the Plugin should produce metadata Events represented by IEventMetadata (declared in the manifest and reported the same way as Object Metadata). Use Plugin Diagnostic Events only when there is an error-like condition, or some event of a technical nature. 

Properties

Plugin Diagnostic Event has a number of properties that can be used in Rule conditions when assigning Actions to the Events.

The following properties of the Plugin Diagnostic Event are automatically assigned by Nx Witness Server:

• Id of the Analytics Engine which raised the Event.
• Timestamp of the moment when the Event was raised.
• Id of the Device (Camera) on which the Plugin was working, in the case the Event was raised by a DeviceAgent rather than an Engine.

The following properties can be assigned by the Plugin when raising the Event:

• Level: either Error, Warning, or Info. It defines the appearance (like color and icon) of the notification.
  • "Error": Something went wrong in the Plugin or its backend, which affected the Plugin's ability to perform its job, either permanently (e.g. until Server restart), or for the particular operation (e.g. given video frame).
  • "Warning": Something went wrong, but it did not immediately prevent the Plugin from doing its job: like temporary loss of connection to a backend, or low free storage space, or some situation from which the Plugin recovered but decided that the user should be informed.
  • "Info": Something unusual or significant happened, not necessarily undesired, and required informing the user or taking some other action (which can be assigned to the Event by the VMS administrator).
• Caption: Brief description of what happened. May omit essential details.
• Description: Detailed information about the event; may include certain technical details (like some ids clarifying the event cause) inside its text. 

As for the Caption and Description, there is no need to include the reference to the Plugin or its Engine — it will be taken from the respective property. But if the Event was produced with respect to a certain Device (Camera), its reference will not be shown to the user and, if needed, must be included in the Caption or Description text. Also, there is no need to repeat in the Description any information that is present in the Caption. 

How to Raise

Currently, the SDK offers to raise Plugin Diagnostic Events, from an Engine or a DeviceAgent of an Analytics Plugin, by calling one of the following methods of an IHandler object supplied to the Plugin by Nx Witness Server:

nx::sdk::analytics::IEngine::IHandler::handlePluginDiagnosticEvent()
nx::sdk::analytics::IDeviceAgent::IHandler::handlePluginDiagnosticEvent()

Each of these methods expects a newly created instance of nx::sdk::IPluginDiagnosticEvent. The most convenient way of constructing this object is using the helper class nx::sdk::PluginDiagnosticEvent.

Note: In future versions, we plan to allow raising Plugin Diagnostic Events via an instance of nx::sdk::IUtilityProvider supplied by Nx Witness Server to each Plugin (any kind).

If the Analytics Plugin implements Engine and/or DeviceEngine via the helper classes nx::sdk::analytics::Engine and/or nx::sdk::analytics::DeviceEngine, it may raise a Plugin Diagnostic Event simply by calling their protected methods:

pushPluginDiagnosticEvent(
    nx::sdk::IPluginDiagnosticEvent::Level level,
    std::string caption,
    std::string description)

When raising Plugin Diagnostic Events, try to carefully plan how often an Event will be raised. Because the Events are registered in the Nx Witness Event-Rule-Action subsystem and typically lead to some kind of user notification, it is not a good idea to raise such Events many times per second (e.g. on each video frame).

For example, if an Event indicates that some undesired situation took place, it is a good practice to raise an Event once on the first occurrence of the situation by setting a dedicated flag variable, and resetting this variable only on some rare occasion like restarting video analytics for the particular Device (Camera), i.e. recreating the DeviceAgent.