Troubleshoot an unresponsive Media Server

Sometimes the Media Server becomes unresponsive. This article explains how to recognize an unresponsive Server and how to collect the logs and memory dumps that Nx Support needs to investigate the issue.

Symptoms

Your Media Server might be unresponsive if you notice any of the following:

• The Server appears to be running, but it doesn't respond to commands.
• You can't reach the Server through the Desktop Client or Server Web Admin.
• Some features stop responding while others continue to work normally.
• Restarting the Media Server resolves the issue temporarily, but the problem returns.

Before you begin

• Collect the logs and memory dumps before you restart the Server. A restart clears the unresponsive state, and the dumps then no longer show the problem.
• Make sure the drive where you save the dumps has enough free space. A memory dump is about as large as the memory the Server process is using, which can be several gigabytes.
• On Linux, you need sudo rights on the Server.

Document the reproduction steps

Write down the exact steps that reproduce the issue. Include as much detail as possible, such as the actions you took right before the Server stopped responding.

Collect the Media Server logs

Compress and save the contents of the log folder for your operating system. To find the log folder, see How to change software logging level and how to get logs.

Create memory dumps

Create two memory dumps of the running Server process, a few minutes apart. Comparing the two dumps helps developers analyze the unresponsive state.

Caution: The Server process pauses while the dump file is written. This can take several minutes and can cause a short gap in recording.

Windows

1. To open Task Manager, press Ctrl+Shift+Esc.
2. If Task Manager shows the compact view, click More details.
3. In the list of processes, find Nx Witness Media Server.
4. Right-click the process, and then select Create dump file.
5. Write down the file path shown in the confirmation message.
6. Wait 5 to 10 minutes, and then try to trigger the unresponsive behavior again. For example, try to rename the Server in the Resource Tree.
7. To create the second dump, repeat steps 3 to 5.

Linux

1. Open a terminal on the Server.

2. If the GNU Debugger (gdb) isn't installed yet, install it. The gcore and gdb tools in the next steps are part of this package.

   sudo apt install gdb

3. Find the process ID (PID) of the Media Server:

   pidof mediaserver-bin

4. Create the memory dump. Replace <folder> with a folder that has enough free space, and <PID> with the process ID from the previous step:

   sudo gcore -o <folder>/mediaserver-dump <PID>

   The dump file is saved as <folder>/mediaserver-dump.<PID>. Creating the file can take several minutes.

5. Extract the stack trace from the dump. Replace <DumpFilePath> with the full path to the dump file:

   echo "t apply all bt 25" | sudo gdb /opt/networkoptix/mediaserver/bin/mediaserver-bin <DumpFilePath> > report.gdb-bt 2>&1

   The stack trace is saved as report.gdb-bt in the current folder.

6. Wait 5 to 10 minutes, and then try to trigger the unresponsive behavior again. For example, try to rename the Server in the Resource Tree.
7. To create the second dump and stack trace, repeat steps 4 and 5. In step 4, use a different file name, for example mediaserver-dump2. In step 5, save the output as report2.gdb-bt so it doesn't overwrite the first report.

Note: If you run gcore without the -o option, it saves the dump as core.<PID> in the folder you run the command from.

Restart the Server and submit the data

1. Restart the Media Server to restore service temporarily.
2. Upload the log folder, both memory dumps, and, on Linux, both .gdb-bt reports to a cloud storage service, such as Google Drive, Microsoft OneDrive, or WeTransfer.
3. Send the download link and your reproduction steps to your reseller. If you're a reseller, share them with Nx Support through a support ticket.

Related articles

• How to change software logging level and how to get logs
• Collect crash dumps for the Media Server and Desktop Client applications