Keep getting network issues on cameras

In Progress

Comments

24 comments

  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    Can you tell us what kind of network issues you encounter? Please also check our support article about Network Issues, perhaps this clarifies what happens in your case.

    Also, in general, you can't compare VLC and Nx Witness completely, since VLC pulls one stream with the UDP protocol and Nx Witness pulls two streams with the TCP protocol.

    Can you explain more about this:

    I am using the stream as well to do other processes before sending it through to the VMS so I have an input stream that I use and manipulate and once that's done I send the same stream out with a few changes to frames.

    It might be useful to check the communication between the device and Nx Witness with Wireshark to have a better understanding what happens between both devices.

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    Thanks for your quick reply

    It just says network issue on the notification bar on the right and there is a pulsating orange box around the camera. Also the camera freezes for a second or two then comes back. 

    Can you tell us what kind of network issues you encounter?

    Ok

    Also, in general, you can't compare VLC and Nx Witness completely, since VLC pulls one stream with the UDP protocol and Nx Witness pulls two streams with the TCP protocol.

    Sure, I take the feed and I grab each frame from it and use an AI to detect any vehicles on it. If it finds one it draws a box around it, then I send the frame back. So I get a normal stream and send out the same stream but with boxes where vehicles are

    Can you explain more about this:

    I have tried Wireshark but I am not sure I am doing it right. Do I use the ip of the cameras and test it against Nx Witness or my ip?

    It might be useful to check the communication between the device and Nx Witness with Wireshark to have a better understanding what happens between both devices.

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    Regarding:

    Sure, I take the feed and I grab each frame from it and use an AI to detect any vehicles on it. If it finds one it draws a box around it, then I send the frame back. So I get a normal stream and send out the same stream but with boxes where vehicles are

    My assumption is that pulling the stream from the camera to the AI application ánd to the VMS is taking too much load on the camera, causing it to freeze every once in a while due to full buffers.

    What typically is done, that there is a device running the AI application, analyzing the camera stream(s), and forwarding the camera stream to the VMS.

    Like this:

    Camera --> AI application --> VMS

    Not sure about the application you're using, but instead of analyzing every frame, it might be useful to analyze fewer frames, like every X frames, instead of every frame. This should reduce the load significantly.

    Also, it might be useful to analyze the secondary low res stream, instead of the main stream, for many AI applications, this is just fine. Again, this should reduce the load significantly. 

    Regarding:

    I have tried Wireshark but I am not sure I am doing it right. Do I use the ip of the cameras and test it against Nx Witness or my ip?

    You could check THIS support article about how to use Wireshark to capture the relevant data.

    Depending on your setup, it might be useful to capture the data between camera and AI device (if you can run Wireshark on that device) and/or the data between VMS and RTSP stream.

    A workaround could be to change the RTP transport stream from TCP to UDP since UDP is more forgiving for such scenario, although you preferably want to keep using TCP for forensic systems.

    You can change this in the Camera Settings menu, tab Expert at RTP transport (1):

     

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    Ok thanks for the advise will test now and see if anything works.

    Just had a look at the transport type and its set to auto and if I change it to UDP, then it runs for a few seconds and then camera goes offline. Then a few seconds later it reconnects back. So looks like UDP wont work.

    Its difficult to test with WireShark between the ai application and the vms as the ai application is on a remote device which I have to ssh into and is on a screen (no gui)

    I'll see if there is a way I can modify it to only detect every nth frame and see how that performs.

    I do want to see if I can view the data between the VMS and the rtsp streams as that will indicate to me what the issue might be.

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    If, after enabling UDP the camera goes online and offline every N seconds, it is likely a resource issue.

    I think you're asking too much from the AI device or camera.

    You wrote, that the AI application is on a remote device, is this remote device on the same network, or is it remote entirely, and outside the network?

    If the configuration is different (situation C), please let me know.

    To Wireshark the issue, it of course depends on the network configuration, but in case of situation A, you use the internal IP address of the AI device as the host filter, and in situation B, you use the external IP address of the AI device as the host filter.

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    The cameras are on the same network as the AI device - so situation A.

    But where would I run Wireshark as I cant open it on the AI device and I don't have access to a machine on that network. I can only ssh into the AI device and I've got the VMS on my machine. This is where it is tricky

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria

    In that case you can run Wireshark on the VMS server.
    If this is a headless Linux server, you can use TCPDUMP, which is a similar application, but for the commandline.

    The following command could be used:

    sudo tcpdump -i <NIC> host <ipAddressOftheAIdevice> and (port 80 or port 554) -w <anyFileName>.pcapng

    For example:

    sudo tcpdump -i eth0 host 192.168.178.23 and (port 80 or port 554) -w test.pcapng
    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    Ok tried it now but what is NIC? and can I find it? I can see that google says Network Interface card, is that it?

    tcpdump: eth0: No such device exists
    (SIOCGIFHWADDR: No such device)

    Above is the error msg I get

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Ok nevermind I found out how to get it, mine is eno1 instead of etho0
    Busy testing it and will get back to you

     

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Norman - Nx Support 

    Sorry what exactly am I looking for? 

    Do I let it run until it finishes or do I stop it and then see the packets sent and received?

    Should I test both ports?

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    When you run Wireshark, you wait till the issue appears and stop Wireshark and you analyse the data you captured with Wireshark to see if you can find a reason for the delay?

    Does the device respond timely to the server requests? Are there many retransmissions? Are there any HTTP code replies? What are they? Etc.

    Once you captured the issue with Wireshark, feel free to share the file with us to check. For usability, please limit the maximum file size to 500 MB max, you can do this with the ring buffer as described in the support article we shared before.

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    I was running this command that you sent me:

    sudo tcpdump -i eno1 host 192.168.178.23 and port 80 -w test.pcapng

    When I ran it, it didnt do anything until I stopped it and then it gave me 2 line of code which said:

    0 packets received by filter
    0 packets dropped by kernel

    I think Im getting confused as I cant run the wireshark application on the headless server and I thought this would do a similar thing.

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    I would be surprised if the IP address I provided in the example is the one that applies to your camera.
    Also, in such case port 554 is important to include as well, since all video uses port 554, unless you changed in the camera.

    sudo tcpdump -i eno1 host <ENTER THE IP ADDRESS OF THE CAMERA HERE> and (port 80 or port 554) -w test.pcapng

    This command will do the same as Wireshark, but from the commandline. The output will 100% identical.

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    yes I did put the correct ip. This is the exact command I am running.

    sudo tcpdump -i eno1 host 196.192.~~~.~~ and port 80 -w test.pcapng

    Should I run it like this then?

    sudo tcpdump -i eno1 host 196.192.~~~.~~ and (port 80 or port 554) -w test.pcapng
    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    If I run it like above I get a syntax error because of the brackets

    -bash: syntax error near unexpected token `('
    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Also you said I should run it when I get the errors and then stop it correct?

    Previously when I tried that with just either one port or the other it gave me the same results as above

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    Sorry, it should have been without the brackets like this:

    sudo tcpdump -i eno1 host 196.192.179.58 and port 80 or port 554 -w test.pcapng

     

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    Yes I tried that yesterday and left it to run for a good 30min to an hour and I still only got

    4 packets received by filter
    0 packets dropped by kernel

    the problem did occur during those 30min, so I would of seen something on here right??

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    4 packets in 30 minutes doesn't seem right.

    I just captured a few seconds (<5) and already got 588 packets.

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    Ok maybe I have the wrong IP, let me try with another and get back to you

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    Ok so I had the wrong ips and am getting the correct ones now, however I've discovered that the cameras are being streamed over WAN so frames will be dropped and also we have another AI device running with those cameras and there is no issue with that one.

    What then could be the issue if the other AI device runs fine. Could it be how the stream is being sent? 

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Something else to think about.

    Both AI devices were working perfectly on an older version of the VMS (v4.2.0.34861)
    but ever since the update (v5.0.0.35272) its been giving us this issue.

    Maybe a new feature was added or something changed that caused this?

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Codi de Faria,

    Do I understand it correctly that multiple applications send requests to one device?

    Like situation C?

     

    If that is the case, I would understand the issue as you described, since most cameras cannot process such amount of requests.

    To continue it is very important to have a full understanding of the system. Which devices are communicating with what devices.

    For instance if multiple devices send different profiles to the camera(s), conflicting settings will occur with such behavior as a result. An option could be to go to thw Camera Settings menu (1), tab Expert (2) and enable Keep camera stream and profile settings (3).

    When this is set, the Quality and frame rate (FPS) settings in the Recording Schedule will become irrelevant and will be used as they are offered by the camera or AI device(s).

    0
    Comment actions Permalink
  • Avatar
    Codi de Faria

    Hi Norman - Nx Support

    This is what I can see if I go into the camera settings on the expert tab. I dont have any option for "keep camera stream and profile settings"

    So the camera just passes its stream which yes does get used my multiple devices then those devices send out another rtsp stream slightly different to the incoming stream (as it has boundry boxes drawn on) to the VMS.

    What bothers me the most is that this was working perfectly on a previous version but ever since the update its stopped working properly.

    0
    Comment actions Permalink

Please sign in to leave a comment.