Skip to main content

Recommendations for Large install 128 plus Cameras

Completed
Jeremy Stantiall

I have an existing installation at a site we manage the IT. But I think it has not been implemented in the best way. We have one NX server. It is a VM running on ESXi 6.7 with 16 CPU's assigned and 32GB of RAM. Host is a HPE SimpliVity cluster, ProLiant DL325 Gen10s. AMD EPYC 7401P 24 Core Processor. System disk is on the SimpliVity storage SSD. The Camera storage is on a Synology RS4021xs+ RAID6 with 15 x 16TB Synology HAT5300 drives + Hot Spare. 2 NFS Volumes presented to VSphere then attached as 4 drives to the VM across 2 SCSI controllers. 

The majority of the camera's are Dahua DH-IPC-HDW4631EMP-0280B's recording at 20FPS on medium quality, Motion + Lo-Res 5.7Mbit/s bitrate. H264 Codec 3072x2048 Res Primary and 640x480 secondary.

The issue is basically constant Not enough HDD/SSD speed alerts across the 4 storage locations. CPU and Memory usage of the server is fine, around 35 and 50% respectively. The NAS also appears to be idling. 

Viewing the streams both live and older recordings seems fine but I am concerned about the alerts. Murphy's law means the frame we need will be the one that is dropped.

Basically I would like to know if I will gain anything by creating more VM's and splitting the Camera load across the servers. And if we do this how would it change the client usability? Would staff need to connect to the correct server to view footage for a specific camera for example. 

Thanks

 

 

0

Comments

5 comments

Date Votes
  • Jeremy Stantiall

    0
  • Anderson Chang
    • Network Optix team

    Hi Jeremy Stantiall,

    Please refer to this support ARTICLE for the information about the "Not enough HDD/SSD speed for recording to" event log.
    Since the message appears constantly, please follow the below instructions and provide detailed information for further investigation.

    1. Check the firmware version of the Nx Server
    The latest Nx Witness version is 4.2.0, if your Nx Witness is not running the latest version, you could first try to update the Nx Witness to the latest release.
    Check: https://my.networkoptix.com/#/download
    Please also make sure the Windows and the Synology NAS are running on the latest firmware.

    2. Camera numbers and the recording bitrate
    - How many cameras connect to the server?
    - What is the bitrate of the primary and secondary stream?

    3. Client and the camera viewing number
    - How many Nx Clients will connect to the Nx Server at the same time?
    - What are the average numbers of cameras being viewed on Nx Client simultaneously?
    - Does the Nx Client often review the video archive?

    4. Networking and storage environment 
    - What is the networking link rate between the server and the NAS?
    - You mentioned 15 hard disks were used to construct the RAID6, how many of them was set to hot spare?

    5. Windows and Synology NAS logs
    Please check whether there are any errors or warnings listed on the Windows Event Viewer and Synology Log Center, share the logs with us if you find anything unusual.

    6. Other software and functions
    - Are there any other software or functions which will consume the networking and storage resource?

    7. Set Nx Server log level to DEBUG, and enable the Performance Alarm on the Synology NAS
    Here are the support articles for your reference:
    Setting DEBUG log level on Nx Server: How to change software logging level and how to get logs
    Setting Performance Alarm on Synology NAS: https://kb.synology.com/en-ph/DSM/help/DSM/ResourceMonitor/rsrcmonitor_settings?version=6

     

    About your questions:

    Basically I would like to know if I will gain anything by creating more VM's and splitting the Camera load across the servers. And if we do this how would it change the client usability? Would staff need to connect to the correct server to view footage for a specific camera for example. 

    It depends on the scenarios and what issue we want to solve. In your case, all of the storage volumes are on the single NAS, I do not think adding VMs on the same physical server could resolve the issue, since the VMs are still accessing the same NAS via the same connection.

    Adding VMs and setting up the failover across the Nx Servers do increase the reliability of the entire Nx system. However, if the VMs run on the same physical server, this can only prevent the video footage lost if any Nx Server or VM crashes, does not work if the physical server fails.

     

    0
  • Jeremy Stantiall

    Hi Chang,

    1. Server is running 4.2.0.32840

    2. 111 Camera's recording, there are some thermal kiosk's on top of this. Majority of the cameras are set to 20 FPS, Medium Quality Motion + Lo-Res. Bitrate 5.7Mbit/s. Primary H264 Res 3072X2048 Secondary H264 Res 640x480 Bitrate 240Kps, FPS 7. 

    3. 2 Viewing stations on full time Live streams only 14 Cams. 5 incident lookup clients. Rarely used it would be extremely unlikely for more than 1 of these to looking for an incident at once.

    4. Unifi Network Switches. Each SimpliVity/ESXi node connected via 10Gbps Fibre to US-16-XG switch with redundant 10Gbps Fibre link to second US-16-XG. Synology has 10Gbps copper connection to each of the US-16-XG switches, Active passive Bond.

     

    I believe it is an MTU issue. In gathering this information found that the Jumbo frames where not enabled everywhere required. Unfortunately enabling it caused some issues with the VM so we have rolled back for now. Will revisit next week. 

     

    0
  • Anderson Chang
    • Network Optix team

    Hi Jeremy Stantiall,

    Thanks for the information update.

    Here is one more thing that would need your help: test the network throughput of the Nx Server.
    You may test the throughput by iperf3, a lightweight network speed test tool: https://iperf.fr/iperf-download.php
    You will need another computer except the Nx Server to test the throughput.
    1. Download iperf3 on the Nx Server and the other computer.
    2. On one of the computers, execute the below command to act as an iperf3 server:

    iperf3.exe -s

    3. On another computer, execute the below command to test the upload speed:

    iperf3.exe -c <iperf3 server IP address> -P 5 -t 30

    To test the download speed:

    iperf3.exe -c <iperf3 server IP address> -P 5 -t 30 -R

    4. Share the result with us.

     

    Besides, if you do not find any issue regarding MTU and jumbo frame in the network yet the issue persists, it is likely we need to have direct access to your server to have further investigation.
    If you agree to provide direct access for us, I will convert this post into our ticket system for you to provide sensitive information.

    0
  • Anderson Chang
    • Network Optix team

    Hi Jeremy Stantiall,

    We have not heard back from you in a while, so we are going to assume your question has been answered and set this topic to 'Completed'.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post