Keep getting network issues on cameras
Completed
Hi Nx
Please can someone help me, Ive been trying to resolve this issue for a while now and just cant seem to get it working.
Sorry in advance, I am new too this so I might get some details wrong here.
I have cameras connected via rtsp which I am getting the feed from them however every few seconds I will get the danger icon by their name and on the notifications It says Network Issue.
I thought maybe it was somewhere on how I am sending the feed but when I VLC using the rtsp url, it works perfectly. No sudden hitches or stops, just smooth video.
I am using the stream as well to do other processes before sending it through to the VMS so I have an input stream that I use and manipulate and once that's done I send the same stream out with a few changes to frames.
So here's the thing when I compare the codecs if the input stream and the stream that I send out its exactly the same except for the decoded format. For the input stream the decoded format is left blank but for the output stream its Planar 4:4:4 YUV.
Is this what is causing the issue?
Sorry if this sounds confusing, im more than happy to re explain.
Thanks
-
Hi Codi de Faria,
Can you tell us what kind of network issues you encounter? Please also check our support article about Network Issues, perhaps this clarifies what happens in your case.
Also, in general, you can't compare VLC and Nx Witness completely, since VLC pulls one stream with the UDP protocol and Nx Witness pulls two streams with the TCP protocol.
Can you explain more about this:
I am using the stream as well to do other processes before sending it through to the VMS so I have an input stream that I use and manipulate and once that's done I send the same stream out with a few changes to frames.
It might be useful to check the communication between the device and Nx Witness with Wireshark to have a better understanding what happens between both devices.
-
Thanks for your quick reply
It just says network issue on the notification bar on the right and there is a pulsating orange box around the camera. Also the camera freezes for a second or two then comes back.
Can you tell us what kind of network issues you encounter?
Ok
Also, in general, you can't compare VLC and Nx Witness completely, since VLC pulls one stream with the UDP protocol and Nx Witness pulls two streams with the TCP protocol.
Sure, I take the feed and I grab each frame from it and use an AI to detect any vehicles on it. If it finds one it draws a box around it, then I send the frame back. So I get a normal stream and send out the same stream but with boxes where vehicles are
Can you explain more about this:
I have tried Wireshark but I am not sure I am doing it right. Do I use the ip of the cameras and test it against Nx Witness or my ip?
It might be useful to check the communication between the device and Nx Witness with Wireshark to have a better understanding what happens between both devices.
-
Hi Codi de Faria,
Regarding:
Sure, I take the feed and I grab each frame from it and use an AI to detect any vehicles on it. If it finds one it draws a box around it, then I send the frame back. So I get a normal stream and send out the same stream but with boxes where vehicles are
My assumption is that pulling the stream from the camera to the AI application ánd to the VMS is taking too much load on the camera, causing it to freeze every once in a while due to full buffers.
What typically is done, that there is a device running the AI application, analyzing the camera stream(s), and forwarding the camera stream to the VMS.
Like this:
Camera --> AI application --> VMS
Not sure about the application you're using, but instead of analyzing every frame, it might be useful to analyze fewer frames, like every X frames, instead of every frame. This should reduce the load significantly.
Also, it might be useful to analyze the secondary low res stream, instead of the main stream, for many AI applications, this is just fine. Again, this should reduce the load significantly.
Regarding:
I have tried Wireshark but I am not sure I am doing it right. Do I use the ip of the cameras and test it against Nx Witness or my ip?
You could check THIS support article about how to use Wireshark to capture the relevant data.
Depending on your setup, it might be useful to capture the data between camera and AI device (if you can run Wireshark on that device) and/or the data between VMS and RTSP stream.
A workaround could be to change the RTP transport stream from TCP to UDP since UDP is more forgiving for such scenario, although you preferably want to keep using TCP for forensic systems.
You can change this in the Camera Settings menu, tab Expert at RTP transport (1):
-
Ok thanks for the advise will test now and see if anything works.
Just had a look at the transport type and its set to auto and if I change it to UDP, then it runs for a few seconds and then camera goes offline. Then a few seconds later it reconnects back. So looks like UDP wont work.
Its difficult to test with WireShark between the ai application and the vms as the ai application is on a remote device which I have to ssh into and is on a screen (no gui)
I'll see if there is a way I can modify it to only detect every nth frame and see how that performs.
I do want to see if I can view the data between the VMS and the rtsp streams as that will indicate to me what the issue might be.
-
Hi Codi de Faria,
If, after enabling UDP the camera goes online and offline every N seconds, it is likely a resource issue.
I think you're asking too much from the AI device or camera.
You wrote, that the AI application is on a remote device, is this remote device on the same network, or is it remote entirely, and outside the network?
If the configuration is different (situation C), please let me know.
To Wireshark the issue, it of course depends on the network configuration, but in case of situation A, you use the internal IP address of the AI device as the host filter, and in situation B, you use the external IP address of the AI device as the host filter.
-
The cameras are on the same network as the AI device - so situation A.
But where would I run Wireshark as I cant open it on the AI device and I don't have access to a machine on that network. I can only ssh into the AI device and I've got the VMS on my machine. This is where it is tricky
-
In that case you can run Wireshark on the VMS server.
If this is a headless Linux server, you can use TCPDUMP, which is a similar application, but for the commandline.The following command could be used:
sudo tcpdump -i <NIC> host <ipAddressOftheAIdevice> and (port 80 or port 554) -w <anyFileName>.pcapng
For example:
sudo tcpdump -i eth0 host 192.168.178.23 and (port 80 or port 554) -w test.pcapng
-
Ok tried it now but what is NIC? and can I find it? I can see that google says Network Interface card, is that it?
tcpdump: eth0: No such device exists
(SIOCGIFHWADDR: No such device)Above is the error msg I get
-
Sorry what exactly am I looking for?
Do I let it run until it finishes or do I stop it and then see the packets sent and received?
Should I test both ports?
-
Hi Codi de Faria,
When you run Wireshark, you wait till the issue appears and stop Wireshark and you analyse the data you captured with Wireshark to see if you can find a reason for the delay?
Does the device respond timely to the server requests? Are there many retransmissions? Are there any HTTP code replies? What are they? Etc.
Once you captured the issue with Wireshark, feel free to share the file with us to check. For usability, please limit the maximum file size to 500 MB max, you can do this with the ring buffer as described in the support article we shared before.
-
I was running this command that you sent me:
sudo tcpdump -i eno1 host 192.168.178.23 and port 80 -w test.pcapng
When I ran it, it didnt do anything until I stopped it and then it gave me 2 line of code which said:
0 packets received by filter
0 packets dropped by kernelI think Im getting confused as I cant run the wireshark application on the headless server and I thought this would do a similar thing.
-
Hi Codi de Faria,
I would be surprised if the IP address I provided in the example is the one that applies to your camera.
Also, in such case port 554 is important to include as well, since all video uses port 554, unless you changed in the camera.sudo tcpdump -i eno1 host <ENTER THE IP ADDRESS OF THE CAMERA HERE> and (port 80 or port 554) -w test.pcapng
This command will do the same as Wireshark, but from the commandline. The output will 100% identical.
-
yes I did put the correct ip. This is the exact command I am running.
sudo tcpdump -i eno1 host 196.192.~~~.~~ and port 80 -w test.pcapng
Should I run it like this then?
sudo tcpdump -i eno1 host 196.192.~~~.~~ and (port 80 or port 554) -w test.pcapng
-
Hi Codi de Faria,
Sorry, it should have been without the brackets like this:
sudo tcpdump -i eno1 host 196.192.179.58 and port 80 or port 554 -w test.pcapng
-
Yes I tried that yesterday and left it to run for a good 30min to an hour and I still only got
4 packets received by filter
0 packets dropped by kernelthe problem did occur during those 30min, so I would of seen something on here right??
-
Hi Codi de Faria,
4 packets in 30 minutes doesn't seem right.
I just captured a few seconds (<5) and already got 588 packets.
-
Ok maybe I have the wrong IP, let me try with another and get back to you
-
Ok so I had the wrong ips and am getting the correct ones now, however I've discovered that the cameras are being streamed over WAN so frames will be dropped and also we have another AI device running with those cameras and there is no issue with that one.
What then could be the issue if the other AI device runs fine. Could it be how the stream is being sent?
-
Hi Codi de Faria,
Do I understand it correctly that multiple applications send requests to one device?
Like situation C?
If that is the case, I would understand the issue as you described, since most cameras cannot process such amount of requests.
To continue it is very important to have a full understanding of the system. Which devices are communicating with what devices.
For instance if multiple devices send different profiles to the camera(s), conflicting settings will occur with such behavior as a result. An option could be to go to thw Camera Settings menu (1), tab Expert (2) and enable Keep camera stream and profile settings (3).
When this is set, the Quality and frame rate (FPS) settings in the Recording Schedule will become irrelevant and will be used as they are offered by the camera or AI device(s).
Please sign in to leave a comment.
Comments
24 comments