Custom time window for Device Disconnect and network issues
Completed
I like a feature where I can finetune how long a device can have network problems before an alert is triggered in NX.
Here is an example of how this is done in Domotz:
This way I can filter out shot dropouts, like when cameras are set to do a dayly reboot.
But still be alerted of serious errors.
-
Hi Fredrik Ahlsen and Patrick Kelly,
The current default time before a network issue show is 10000 ms.
I think this a quite a fair number and I wonder why it should be more and if so how much? I would rather be in favor of lowering this limit. Why not 2000 ms?I assume you both know you can change the number of actions you receive from such events in the Rules Engine?
The default rules are:
The interval of action for a Camera Disconnected notification is by default 30 seconds and for an email 6 hours.
The interval of action for a Network Issue notification is by default instant (actually 10 sec after the event due to the threshold) and for an email 6 hours again.You can change this number to your liking; anything between instant and 9999 days.
But why not resolve the issue, instant of ignoring or hiding the issue?
I encountered the Network Issue quite a few times now and nearly without any exception it could be routed back towards the cameras. The few exceptions were VLAN/VPNs or other advanced network issues.
To understand a bit more about the way a VMS works, opposed to a NVR for example is that the VMS in general runs on a more powerful and versatile piece of hardware than the average NVR. This enables you to manage high frame rates, big resolutions and minimal compression.
But, often with a lower price in mind, quite a few popular brands, and I could name them but I won't, aren't capable of delivering these request due to too limited hardware resources on the camera side. I also see quite a few brands that never have any of such issues.
It is quite interesting to see such process in Wireshark. You see a continuous stream of communication between VMS server and camera, the TCP 3-way handshake, so for each request the server sends, the camera has to send a confirmation and what you notice when capture this communication is that there are cameras fail to ack(knowledge) these request in time and server will send the request again (retransmission), again, and again until the camera acknowledge the request or when the buffer of the camera explodes, there will be a TCP teardown which give the camera the opportunity to clear its resources and start over again. Often these retransmissions and TCP teardown result in RTP package loss in Nx Witness, not all, but it could be many.
That being said, an RTP package loss now and then is not issue. But frequent packet loss is.
Often it can be resolved by sending smaller packets towards the camera so it is easier to process. You can achieve that as follows;
- Reducing the framerate
- Reducing the compression
- Reducing the resolution
- (Reducing in-camera analytics)
Often we see people aiming a camera at a scene where the goal of the camera is just detection or observation. In general 5 fps @ quality Low will be fine for this purpose. Also, it might be useful to reduce the resolution to the screen resolution. A 1080p stream does the job just fine since the goal of the camera is to see if someone is there and see what he or she is doing to a certain degree.
For more demanding cameras for identification of recognition purposes 15 fps @ quality High is a more sensible option, but might require more powerful cameras that can deliver such stream. Such cameras are in general also more expensive.
I always like to compare it with cars. If you just need a car to bring you downtown from a to b, most cars will do the job, but if you need to drive long distances (high frame rates) at a higher average speed (minimum compression) in the most comfortable way (the highest resolution) you probably buy a Mercedes E class or similar which is obviously more expensive.
Or to keep it closer to our business. Most of us could do their job on an entry level laptop with 4 GB RAM and a Pentium processor, but it might sometimes freeze and therefore most business laptop at least have an i5 with 8 GB RAM.
-
Everything in response makes complete sense, as an engineer. As the loss prevention manager at a Bank with 300 locations, constantly in a push pull with his IT department, Wiresharking the network is not an answer. (as fascinating as it may be). The RTP timeout in the advanced setting of the web client acknowledges that adjustment-based on network is necessary but not having the ability to adjust within the client indicates it is really only meant for an engineer. I mean how many end users are operating in ms, right ? If interval of action was the answer I am not sure why you needed an additional 1000 words in your response, other than you are aware of the issue.
Again from a systems engineer standpoint, your response makes complete sense. Practically speaking (IMO) you are missing the mark.
-
Yes, the network issues are usually caused by the camera.
But to completely solve that problems is almost impossible.
I think that most cameras will have 99,9% uptime. And 0,01% network issues.
To try to get 100% perfect network connection to all cameras would be extremely hard to acheive and probably not something the end client will pay for when you take into account how much work that will be and what it will end up costing.My conclusion is most systems will have some minor network issues.
And the way Nx handle these is just annoying for both the client and the installer, with again lead to that the feature is disabled.
Nx need some tolerance for minor issues, to make it possible to see the critical issues. Right now you can`t see the forrst because of all the trees.
The two options availabe without a lot of extra work boil down to:
1. Get a lot of annoying warnings or 2. get no warnings at all.
I see this happening all the time in real installations.
Installers just disable the network issues warning and very often the disconnect warnings too.
Here is how I think is should work with default settings:Device disconnect: Only show a warning if a device have been offline for more than 2 minutes withour reconnecting.
Network Issues: Only show a warning if the device have had more than XX issues the past 24 hours.This would allow normal things to happen (Auto reboot, firmware updates, minor package drops) without showing a lot of annoying warnings.
In the new System Health monitor, you can display a more detaild list of all warnings for the engineer.
But please keep them out of the client by default. -
Have you had a chance to test the new Health Monitoring feature we introduced in 4.1?
We have plans to add notifications in it, so you will be able to set rules like
When Events: Stream issues count per 24h is more than 10 -> Send email/Raise Alert
We are still discussing if it should be some additional interface near health monitoring in the cloud/web client or we should integrate it somehow in the Event Rules engine.
If it will be in the event rules engine, then Show Desktop notification can be chosen, so your scenario will be fulfilled by counting the number of issues, till it reaches some critical point.
What do you think? -
Hi Patrick Kelly and Fredrik Ahlsen,
I think I do get the mark and I also see where we can make the improvement. I totally understand that it can be challenging to achieve the ideal situation.
To sum it up; don't bother operators with owner/admin level or engineer level notifications right?
Currently, the default rules were set up in a way that the 'annoying' desktop notification are shown to the target 'All Users' in the rules' engine is while it makes sense to target them for 'Owner' and 'Administrators' only.So please tell me, without making any promises or expectations;
What if we keep the default rules as they are, but change the Target from 'All Users' to 'Owner / Administrator', would that be enough for now until the new health monitoring tool is updated?
Would different default intervals be helpful. If so, what should it be?
-
I hate to necro old threads, but - was there every any movement or indication that this would be implemented into NX Witness? I can certainly see the value of allowing a variable timeframe for added cameras - for one, it would cut down on unnecessary email alerts when a camera isn't detecting motion, or (for my country especially) when the internet cuts out intermittently.
Wouldn't go so far as to program in default intervals - make it a slider, or an input field, or something similar.
-
Hi Matthew Mifsud,
Here is how we look at this.
Video/audio from cameras to server is typically (always) transferred via RTP over TCP. Since TCP is a
reliable protocol in theory it should not be any packet lost, at least not on the network layer.Packet loss happens on the application layer: likely the camera decides to drop a packet.
Each RTP packet has a sequence number and the camera puts packets into the "send" buffer. We believe due to some jitter on the network and(!)/or small camera buffers(whatever implementation is), sometimes the camera decided to drop a packet since it did not manage to send data "in time" before "send" buffer is over.
Likely in this scenario, the camera drops multiple packets, till the next frame, and encodes the keyframe (aka I frame) asap.If you see Network issues likely it means, you indeed have some issues in the network or camera buffers cannot handle your network jitter or similar. Likely it means some frames are missed(in your archive you have 29, not 30 frames per second around the network event).
Some camera manufacturers even asked us to "hide it" for their cameras. We did not. In the software, we decided to let people know about the issues rather than masking them. You can observe the same issue(RTP packet loss in Wireshark).
If you want to ignore the issue, you can simply disable the warning by going to the Event Rules and disabling the rule "Network Issue".
Please sign in to leave a comment.
Comments
14 comments