What to do when a cloud-connected system shows unreachable or offline?

Nx Cloud is a major component of Nx Witness and extends the functionality of Nx Witness Systems by providing cloud services hosted on the internet. Nx Cloud allows users to access their Nx System from an external network without enabling port forwarding. Users can connect to their Nx Systems via all of the Nx Witness Clients we offer (Nx Desktop, Nx Mobile, Nx WebAdmin, and the Nx Cloud Portal). 

A cloud-connected Nx System can be accessed via direct IP address if applicable, STUN, or through the Nx mediator, a relay service developed by Nx which is hosted on Amazon Web Services (AWS). While a system is connected to Nx Cloud, the Nx Client will show the connection status (“Offline”, ”Unreachable”, or just a cloud icon means ready to connect).

What does “unreachable” or “offline” mean to a Nx Cloud System?

Offline means the Nx System is not able to communicate with the Nx Cloud and the System is not available at the moment. It could be due to the server not starting, the connection is completely cut off, or the Client is not connected to the internet.

Unreachable means the Nx System is online and in working state, but unable to be connected via the Client through Nx Cloud due to various reasons and may be solved by simple actions depending on the situation.

Most common connection issues Nx Cloud Systems encounter

Firewall Configuration / Networking Restriction

Although Nx Cloud does not need extra configuration to work in a general networking environment, sometimes we see that a required service is blocked by the enterprise/local network firewall.

For example, some companies block outgoing/incoming connections to any URL that wasn’t granted specific permission, preventing necessary communication from Nx Mediator and Nx Relay (See the Cloud Connect article for more details), leading to a loss of connection between the Nx System and Nx Clients. 

Solution:

To solve/confirm this issue, we provide the connection test tool for the users to check their connection availability. The tool would provide the result of connectivity tests to all the publicly trusted services used by Nx Cloud and Nx Witness. Ideally, you should see all connections have successfully passed and the required ports are all open. If any URL in the result is blocked, please talk to your networking technician to allow the traffic to pass through the firewall.

Self-signed Certificates

High security locations sometimes have very strict network policies. Not only will the incoming/outgoing traffic be limited, but devices are required to reside within the enterprise network or use a certified SSL certificate to be able to access the enterprise network.

Now by default, Nx Witness uses self-signed certificates. Sometimes this could lead to the System not being able to access the network in some cases if the Nx Client is on an external network. 

Alternatively, if you upgraded your Nx Witness System from an older version (e.g. 4.2 or before), the self-signed certificate could have expired, so any HTTPS connection will not be established successfully. 

Solution:

1. If the issue is due to the self-signed certificate expiring:
   a. Stop the Nx Server service.
   b. Navigate to the directory for the Nx Server SSL certificate.

   Windows
   C:\Windows\System32\config\systemprofile\AppData\Local<%COMPANY.NAME%><%COMPANY.NAME%> Media Server\ssl

   Ubuntu
   /opt/<%INSTALLER.NAME%>/mediaserver/var/ssl

   c. Delete the default.pem (cert.pem if your version is 4.2 or before) to remove the old certificates
   d. Start Nx Server, and it will create a new certificate with an updated expiration.
2. If you need to use your own SSL or enterprise verified SSL, you would be able to replace the Nx self-signed one to your own one by referring to this article.
   
   NOTE: If the issue is not resolved, please try to delete the certificates (default.pem or cert.pem) again and then restart the machine once more.

Root CA Certificates

If you are using the Windows platform (e.g. Windows 10/11/server/IOT/Server, etc.), sometimes you might encounter an issue where the root CA certificate expired or a lack of the required certificates. This might be causing your system to not establish the connection with AWS correctly. This usually does not affect daily internet browsing, but it could be an issue while you are trying to access some services which force you to use secure connections (HTTPS) or access the service that is run on the public cloud.

Some of our cloud services use HTTPS certificates provided by Let’s encrypt service and there is a known issue that root CA used by Let’s encrypt expired. More details can be found here.

In Nx Witness 5.0, we enabled certificate verification for any outgoing connection from Nx Server to the Nx Cloud services. This security enhancement greatly reduces the chances of man in the middle attack(MITM) on the Nx Server and improves the secure level of our software. 

If you update your system to Nx Witness 5.0, and it unexpectedly becomes unreachable via cloud connection, it is likely that Nx Server faces a certificate verification issue. Nx Server relies on the OS to provide a list of trusted root certificates. If you’re using an older Windows version without the latest updates, it is likely the OS could be missing either some required Root CA certificates or the new root certificate ISRG Root X1(sometimes could be both), and would require you to add it to the system. 

DST Root CA X3 Expiration (September 2021)

In rare cases, you might encounter some unexpected issues while both ISRG Root X1 and DST Root CA X3 are presented in the system, so you may need to delete the expired DST Root CA X3 certificates from your OS.

To quickly check if you are facing this issue, try to access this URL from the exact machine which has Nx Server installed. You are able to use any browser (e.g. Google Chrome, Firefox, Edge, etc.) to see if the HTTPS warning is shown. If you see that the browser shows the security warning, then it is highly likely you need to add ISRG Root X1 certificate to your Windows machine.

Solution A:

1. Search certmgr.exe and launch the application.
2. Navigate to Trusted Root Certification (usually the 2nd node).
3. Delete DST ROOT CA X3
4. Make sure ISRG Root X1 is present. If it is missing, see Solution B.
   

Solution B:

1. Download the latest modern certificates isrgrootx1.der.
2. Open the downloaded certificate and click Install Certificate.
   
   
3. In the Certificate Import Wizard, select Local machine.
   
4. Choose Place all certificates in the following store.
5. Select Trusted Root Certification Authorities and select OK.
   
   
6. After you see the import success dialog, restart the Nx Server, or restart the machine. The issue should be resolved.  

Missing Starfield Root CA certificates

In certain cases, the Windows platform may have fewer built-in Trusted RootCA certificates installed, particularly on the version with strict security control, such as Windows Server 2019. The lack of required certificates may result in an incomplete trust chain, so the SSL validation process may fail when communicating with the Nx Cloud service.

To check if you have the necessary Root CA certificate installed in a certain server, please follow the steps described below:

1. Search certmgr.exe and launch the application.

2. Navigate to Trusted Root Certification (usually the 2nd node).

3. Make sure the Starfield Root Certificate Authority - G2 and Starfield Class 2 Certification Authority are present. If they are missing, do the next step 4 - 9 to install them manually.
   
   

4. Download the certificates from the following link:
   - Starfield Class 2 Certification Authority
   - Starfield Root Certificate Authority - G2

5. Open the downloaded certificates and click Install Certificate one by one.

6. In the Certificate Import Wizard, select Local Machine.
   

7. Choose Place all certificates in the following store.

8. Select Trusted Root Certification Authorities and select OK.
   

9. After you see the import success dialog, please restart the Nx Server application, or restart the machine. Then the certificates will be applied and your issue should be resolved.