Is the Nx Cloud up? Visit our Status Page for the current health and performance of the Nx Cloud.

Status Page

Custom SSL Certificate Not Applied on Nx Media Server 6.1.0.42176

Comments

3 comments

  • Mohammed
    • Rising star (Silver)

    Hi Reeve,

    My first thought is, your mediaserver still has the old cert.pem.

    What's in your C:\Windows\System32\config\systemprofile\AppData\Local\<%VENDOR.NAME%>\<%VENDOR.NAME%> Media Server\ssl folder?

    If you still have both the old and new one in that folder, I suggest moving the old one even if the name isn't the same.

    If that's not the case, you may want to attach the server's logs. Left click Tray assistant icon in the Taskbar > Show Server Log

    0
  • Reeve Hsu

    Hi Mohammed, thank you for your assistance. 

    I have already tried deleting the default.pem file and restarting the Nx Media Server service. However, the default.pemfile is automatically regenerated after the service restarts, and the server continues to use the default self-signed certificate.

    Could you please help investigate why the default certificate is being regenerated and advise me on how to correctly apply our custom certificate?

    I couldn't find a place to upload logs. But I used GPT to consolidate the info and hopefully it helps:

    Nx Witness Media Server — Log Analysis Summary ================================================= SOURCE FILE: main.log (24,684 lines / 3.9 MB) LOG WINDOW: 2026-10-13 09:56:09 to 2026-10-14 10:07:06 (approx. 24 h; the log begins mid-session - the server was already running when the window opens) PREPARED: 2026-10-14 PURPOSE: Key findings extracted from a high-volume log with extensive duplicate messages, condensed for vendor (US) support review. 1. SERVER & ENVIRONMENT ----------------------- Product            : Network Optix Media Server (Nx Witness) Version            : 6.1.0.42176 (revision b58aeb045465) OS / account       : Windows, service runs under systemprofile (Local System account) Virtualization     : VMware VM (vmxnet3 Ethernet adapter) Server HWID        : 66357130-efb9-f58c-ea48-2b889534cc8e Log file           : C:/Windows/system32/config/systemprofile/AppData/Local/Network Optix/Network Optix Media Server/log/main.log Log config         : 10 MB per file / 250 MB volume, level: info Storage            : d:/HD Witness Media/... 2nd server in system: 831e12e8-e8b7-2358-c73c-20389ec80889 @ https://vmscntj006:7001 Cameras seen       : Axis (rtsp://10.95.188-190.x/axis-media/media.amp), Hikvision B310/B410 (ONVIF), Uniview and others Load before restart (10-14 09:46): OS CPU 54.2% / server process CPU 34.9%; OS memory 56.8% (10-13 10:16) 2. SEVERITY OVERVIEW (after deduplication) ------------------------------------------ WARNING : 20,983 lines (85.0%) INFO    :  3,698 lines (15.0%) ERROR   :      3 lines (~0.01%) The 20,983 WARNING lines reduce to 10 distinct message classes (see section 4). The 3 ERROR lines are the most important part of this log (section 3). 3. CRITICAL EVENT — SERVER RESTART ON 2026-10-14 ~10:06 (crash + certificate failure) ------------------------------------------------------------------------------------- TIMELINE (all times local):  - 2026-10-13 09:56:09  Log window begins (server already running)  - ~2026-10-14 10:05:03 Shutdown initiated (inferred - see notes)  - 2026-10-14 10:05:18  Recordings stopped for all cameras (graceful shutdown sequence)  - 2026-10-14 10:05:58.859  ERROR - CRASH: ASSERTION FAILED at media_server_process.cpp:1442: "Server was unable to stop within 55s"  - 2026-10-14 10:06:31.475  Media Server restarted, version 6.1.0.42176  - 2026-10-14 10:06:44.876  Crash dump mediaserver.exe_6.1.0.42176-b58aeb045465-default_12860.dmp uploaded to https://crashes.vmsproxy.com/crashserver/api/report  - 2026-10-14 10:06:44.937  ERROR - SSL: "Failed to read private key from PEM"; cert.pem could not be parsed  - 2026-10-14 10:06:44.952  Server certificate auto-restored from backup default_20260723_092530_031.pem (backup dated 2026-07-23)  - 2026-10-14 10:07:06  Log window ends (cameras reconnecting, streaming resuming) EXACT LOG LINES:  2026-10-14 10:05:58.859   1c04   ERROR CRASH: ASSERTION FAILED: C:\P\573cb5b5af\nx_vms\vms\server\nx_vms_server\src\media_server_process.cpp:1442 (false) Server was unable to stop within 55s  2026-10-14 10:06:31.475   37b4    INFO START: Network Optix Media Server started, version: 6.1.0.42176, revision: b58aeb045465  2026-10-14 10:06:44.876    cf8    INFO ec2::CrashReporter(0x1ce15de0920): Send C:/.../mediaserver.exe_6.1.0.42176-b58aeb045465-default_12860.dmp to https://crashes.vmsproxy.com/crashserver/api/report  2026-10-14 10:06:44.937   37b4   ERROR nx::network::ssl::Certificate: Failed to read private key from PEM.  2026-10-14 10:06:44.937   37b4   ERROR nx::vms::server::SystemCertificate::CertificateInfo(0x2ee0eff280): Failed to parse certificate from file 'C:/.../ssl/cert.pem': Failed to read private key from PEM.  2026-10-14 10:06:44.952   37b4 WARNING nx::vms::server::SystemCertificate(0x1ce15d0f2c0): Restore certificate for Server {66357130-efb9-f58c-ea48-2b889534cc8e} from backup C:/.../backup/ssl/default_20260723_092530_031.pem FACTS vs. INFERENCE:  - Fact: a stop was requested, all recordings stopped at 10:05:18, and the process still had not stopped 55 s later -> assertion + crash dump.  - Inferred: the stop request was at ~10:05:03 (55 s before the assertion). The log does not record who/what requested the stop (planned restart, update, or crash-driven).  - Fact: after restart, cert.pem's private key was unreadable and default.pem was missing; the server restored a certificate from a backup created 2026-07-23 and completed SSL init with it.  - Unknown: the root cause of the cert.pem corruption. 4. DEDUPLICATED WARNING INVENTORY (20,983 lines -> 10 classes) -------------------------------------------------------------- W1. rules::Ec2Router - "Unexpected action received: textOverlay (7,351) / enterFullscreen (6,898) / exitFullscreen (317)"    Count: 14,566  | Window: whole window  | 59% of all warnings    Log flood. Rule engine receives actions it does not expect - typically a client or rule/action misconfiguration. W2. onvif::PtzController - "Wrong PTZ type. Only DevicePtzCoordinateSpace is supported"    Count: 3,618  | Window: 10-13 10:20 to 10-14 09:29    PTZ position polling incompatible with the cameras' PTZ type. W3. onvif::PtzController - "Zero value of Zoom is received"    Count: 2,037  | Window: same as W2    21 cameras (B310/B410 models) report zoom = 0 on position queries. W4. network::ReverseConnectionManager - "N requested connection from {831e12e8...} were never satisfied" (246) + "Incoming connection was not requested from {831e12e8...}" (144)    Count: 390  | Window: 10-13 12:40 to 10-14 09:01    Second server (vmscntj006:7001) unreachable / connection unstable. W5. hikvision::IsapiPtzController - "Only device coordinate PTZ space is supported"    Count: 154  | Window: 10-13 21:01 to 10-14 10:06    Hikvision ISAPI PTZ limitation, recurring across devices. W6. metrics::StorageController - "Skip missing resource"    Count: 67  | Window: 10-14 10:06:39 (startup)    Database references resources that no longer exist. W7. QnRtspConnectionProcessorPrivate / QnRtspDataConsumer - RTSP send() failed, OS code 10035 (WSAEWOULDBLOCK)    Count: 23  | Window: 10-13 10:22 to 10-14 06:59    Streaming stalls: 14 sends blocked > 100 s, max 1,072 s (~17.9 min) at 10-14 06:19:15. W8. onvif::Resource - "Failed to synchronize camera clock to match VMS clock" (camera A4-4C-62-5A-D7-B1)    Count: 2  | Window: 10-13 23:01    Single camera. W9. time::ServerTimeSyncManager - "Can't read time from server 97a95079-...: Can't establish connection to the remote host"    Count: 1  | Window: once    Time server briefly unreachable. W10. network::RequestDelegator - "Failed to connect to Server {831e12e8...}"    Count: 6  | Window: 10-13 14:36 to 22:35    Same root cause as W4. 5. OTHER OBSERVATIONS (INFO level - mostly normal) -------------------------------------------------- - Recording churn, camera A4-14-37-81-D0-9A_channel=2: 386 stop / 378 start events between 22:23:47 and 22:58:04 on 10-13 (~35 min), both Hi- and Low-quality streams. Indicates stream instability on this camera; may correspond to recording gaps in that period. - Server time sync: 103 INFO entries, correction of ~+/-1.2 s every ~10 min. Worth monitoring (typical for VM clock drift), not an error. - Live cache: 159 x "Enabling Live Cache for secondary stream" - normal. - Backup recording: main + backup recorder start/stop events - normal for a system with backup storage. - Module disconnects: 82 x "Removing connection to module {831e12e8...} by URL https://vmscntj006:7001" - consistent with W4/W10. 6. QUESTIONS FOR SUPPORT ------------------------ 1. CRASH (highest priority): analysis of dump ...default_12860.dmp / assertion at media_server_process.cpp:1442 ("Server was unable to stop within 55s"). Is this a known issue in 6.1.0.42176? What typically prevents mediaserver.exe from stopping within the 55 s timeout? 2. CERTIFICATE: what causes "Failed to read private key from PEM" on cert.pem (with default.pem missing)? Is auto-restore from a ~3-month-old backup the expected recovery path, and are there security implications of running on the restored certificate? 3. RTSP STALLS (W7): are send timeouts up to ~18 min (WSAEWOULDBLOCK) a known client/network pattern? Recommended mitigations (client-side, network, or server tuning)? 4. Ec2Router FLOOD (W1): what generates "Unexpected action received: textOverlay / enterFullscreen / exitFullscreen", and what is the supported way to stop the flood? 5. PTZ WARNINGS (W2/W3/W5): with these camera models (Hikvision B310/B410 via ONVIF/ISAPI), are "Wrong PTZ type / Zero value of Zoom / Only device coordinate PTZ space" expected behavior, or a firmware/configuration issue on the camera side? 7. RECOMMENDED CHECKS ON OUR SIDE (parallel to the support case) ---------------------------------------------------------------- 1. Identify what triggered the 10-14 ~10:05 restart (Windows Event Log, planned maintenance, auto-update) - explains the crash context. 2. Verify connectivity and health of server vmscntj006 (831e12e8-...) - W4/W10 point to it. 3. Review event/action rules and active client sessions for textOverlay / fullscreen actions (W1). 4. Check camera A4-14-37-81-D0-9A stream stability (network path, firmware, PoE) - see section 5. 5. Check firmware on the 21 B310/B410 PTZ cameras and their ONVIF profile support (W2/W3). 6. After the certificate restore, confirm the restored certificate is valid, and monitor cert.pem for recurrence. METHOD NOTE: Counts are exact line counts from main.log after masking pointers/GUIDs/camera names; duplicate and "TOO MANY SIMILAR MESSAGES" suppression lines are included in their class totals. Times are as recorded in the log (local time of the server).

    0
  • Mohammed
    • Rising star (Silver)

    Hey Reeve,

    I can already see few abnormal lines :

    SSL: "Failed to read private key from PEM"; cert.pem could not be parsed  - 2026-10-14 10:06:44.952  Server certificate auto-restored from backup default_20260723_092530_031.pem

    2026-10-14 10:06:44.937   37b4   ERROR []: Failed to read private key from PEM.

    2026-10-14 10:06:44.937   37b4   ERROR []: Failed to parse certificate from file 'C:/.../ssl/cert.pem': Failed to read private key from PEM.

    2026-10-14 10:06:44.952   37b4 WARNING []: Restore certificate for Server {66357130-efb9-f58c-ea48-2b889534cc8e} from backup C:/.../backup/ssl/default_20260723_092530_031.pem FACTS vs. INFERENCE:  - Fact: [...]  - Fact: after restart, cert.pem's private key was unreadable and default.pem was missing; the server restored a certificate from a backup created 2026-07-23 and completed SSL init with it.  - Unknown: the root cause of the cert.pem corruption.

    It sounds like the mediaserver can't find the private key in the .pem file. Either because the file itself isn't present (unlikely), or its content isn't what the mediaserver expects. Which then causes it to go find a backup .pem and restore it hence why you keep seeing the default cert

    Two things I'd like to know:

    1. What's the content of your .pem file (The one you generated)? Is it :

    -----BEGIN CERTIFICATE-----
    [CERTIFICATE]
    -----END CERTIFICATE-----
    -----BEGIN PRIVATE KEY-----
    [PRIVATE KEY]
    -----END PRIVATE KEY-----

    2. Is this a self-signed certificate?
     

    Mohammed

     

    0

Please sign in to leave a comment.