What is a Network Issue?

What is a Network Issue?

When Nx Server receives RTP packets and determines that packets were lost and/or did not arrive as smoothly (in the correct order) as expected.

By default, this triggers the “Network Issue'' notification in Nx Desktop on the Notification Panel that appears each time a network issue is detected. Several possible descriptors of this notification are listed below:

• No signal/disconnected
• Connection to camera was unexpectedly closed
• No data received for 10 seconds or more
• RTP error in primary stream

Note: The Nx Server log file will show a new entry indicating packet loss.

Network error detection in Nx Witness is very accurate and can detect even minor hiccups on your network that other VMS software would ignore.

Video and audio from cameras to the Server is almost always transferred via RTP over TCP. TCP is reliable and will not lose packets on the network layer but it can occur on the application layer, such as when a camera drops a packet.

Each RTP packet has a sequence number, and the camera puts packets into the "send" buffer. Sometimes the camera decides to drop a packet since it did not manage to send data in time before the "send" buffer is over due to

• jitter on your network,
• inadequate camera buffers (either due to too small of a size or simply a poorly implemented buffer by the manufacturer),
• or both (camera buffers cannot handle your network jitter).

As a result, the camera drops multiple packets until the next frame, and encodes the keyframe (i.e. I frame) as soon as possible. Some frames are missed (e.g. in your archive you have 29, not 30 frames per second during the time the network event occurred).

If the sensitivity of the network error detection in Nx Witness is not to your liking, you can disable the warning by going to the Event Rules and disabling the rule "Network Issue".

Most common cause of a Network Issue

A network issue is often triggered by a camera or stream that is not being received as expected by Nx Server. 

To troubleshoot this cause, you will need to install Wireshark, filter between Nx Server and the camera, and run a capture. Read How to create a Wireshark capture to learn how to set this up.

A Wireshark capture will allow you to see in detail what is happening in the network between Nx Server and the camera during session initialization, session close, and everything in between. 

There are several issues to look out for in the Wireshark capture or other troubleshooting tools that can capture a camera or stream-related network issue. 

See examples of the most common scenarios to look for and their associated Wireshark capture below:

• The camera failed to initialize with Nx Server. As a result, the camera returns a 401 Unauthorized to the RTSP SETUP request sent by Nx Server.

  

• Nx Server sent three [TCP DUP ACK] after it detected an unstable or underperforming camera that dropped frames, lost packets, or received packets out of order. After a while, the packets were presumed to be lost and the camera responded with "Fast Retransmission" and “Out of Order”.

  

• The camera was concurrently connected to a 3rd party system or interface which  competed for the camera’s streams: This can be confirmed by not isolating the capture between Nx Server and the camera in order to see all the packets the camera is receiving from other sources.
• The camera successfully sent RTP packets for a while but then sent TCP [FIN, ACK] status to Nx Server, initiating TCP session close. Nx Server sent the [ACK] flag and then an RTSP TEARDOWN command. The camera did not process this correctly and abruptly closed the connection, none of which Nx Server expects to see.

  

Other causes of a Network Issue

Network Bandwidth is not adequate:

• The bandwidth between the camera and the server is not sufficient and is causing RTP packet loss, most commonly due to 
• poor connectivity (slow networks) or 
• capacity issues (too much data for existing network infrastructure).

Networking devices that are not configured properly or are incapable of handling the necessary traffic types:

• Some PoE switches do not adequately handle using UDP as the connection protocol between cameras and servers. Networking devices have inherent limitations which may prevent adequate handling of traffic or throughput.
• Two or more devices may have been assigned the same IP address.

An issue with network cabling causing packet loss:

• Cabling runs that are too long
• Use of the wrong type or quality of cabling
• Poor condition of cabling,
• Too many intermediate hops between Nx Server and the device.

Firewalls and/or antivirus applications delay the responses between the Nx Witness Media server and the connected device:

• Set up rules in the firewall or antivirus application to avoid such delays.
• Disable the antivirus application for Nx Server.

More troubleshooting steps you can try:

• Check your network bandwidth. 
  Drag & drop the Server experiencing the issue onto the Viewing Grid in Nx Desktop and monitor the network traffic to look for any obvious issues.
  
  
• Isolate the problem camera/stream by removing devices one by one and reboot the affected camera.
  Check to see if the issue is reproduced. If you are seeing this issue with an RTSP/HTTP stream you may want to remove and add the stream(s) again to see if that resolves the issue.
  
  
• Open the camera streams in VLC (with logging) and check if the packets are lost there as well.
  The stream's URL can be retrieved on the Advanced tab in the Camera Settings dialog (“Primary Stream”, “Secondary Stream”). For additional information, see How to Check a Video’s Bitrate In VLC.

How to avoid a Network Issue in general:

1. Make sure your network is stable and that throughput is adequate between the affected camera and Server.
2. Choose stable, dependable cameras / streams.
3. Simplify the network architecture as much as possible so that the path between the camera/stream and Server is as direct as possible. The more routing you have the more likely a camera/stream is to be affected.
4. Make sure to use high-quality cabling / wireless access points.
5. Change camera settings to reduce bandwidth consumption and packet latency to match the capacity of the network and the overall system.
6. Upgrade camera firmware to the latest version.

Questions

If you have any questions related to this topic, or you want to share your experience with other community members or our team, please visit and engage in our support community or reach out to your local reseller.