Witness / Ubuntu freezing and requiring manual FSCK
Answered
I have multiple custom built Nx Witness servers running on Ubuntu 18.04 that are getting random system freezes requiring a manual reboot. After the reboot, I am having to run FSCK to fix the file system on the OS drive which is a 120 GB SSD (Kingston). Is Witness freezing and the reboot causing the need for the FSCK or is it the other way around? Anyone experiencing anything like this?
-
Official comment
Hi Rich,
I don't have such experience. Actually my experience is the opposite; Ubuntu is really solid. All my headless servers use Ubuntu 18.04 and my desktops run 19.10 (officially not supported) and are really stable.
I would check the ubuntu logs and maybe they reveal the root cause of the freezing.
If you are using the desktop version, you can view them via the Ubuntu dash and search for logs.
If you are using the server version (headless), you can view the logs from the /var/log directory via the following command:$ dmesg|less
In this way, you will find all logs from the /var/log directory which could be a bit too much information.
You can do some filtering with:$ dmesg | grep <keyword>
They <keyword> will be replaced by something you like to look for, like cpu for example. The command will be:
$ dmesg | grep cpu
In this way, all cpu related log entries will be displayed. You can also add the less command to make it easier to search the log files
$ dmesg | grep cpu | less
The dmesg command opens all the logs from the /var/log directory. In order to open the log file from some other location, use the following command:
$ cat <directory>
For example:
$ cat /var/log/syslog
Again you can add the less and grep commands to make your life easier.
Comment actions -
https://support.digital-watchdog.com/kb/article/399-blackjack-fsck-boot-error-ubuntu-linux/
How about now? Have you heard of it now? What is being done about this issue?? The server STOPS recording. This is how companies like ours go out of business. Please let me know what is being done to correct this massive issue. -
Thank you for the link Rich Pedersen.
I'm not sure what you expect from us.
The only information provided to us is that you experience random system freezes with Nx Witness and Ubuntu 18.04.
As mentioned in my previous reply, some further investigation in the OS logs and the server logs is required. As soon as there is something alarming in these logs, we can investigate the root cause (in case of Nx Witness) and if nothing can be found in the server logs, the log level can be elevated from Info to Debug.
In case nothing is found in the logs, there should be a clear reproduction scenario, otherwise there not much for us to investigate and resolving the issue is based on luck with a pinch of magic.
Since you link to the Digital Watchdog site and their specific hardware; in case is concerns the Blackjack NVRs it should be discussed with Digital Watchdog. Also, the article is about the boot process, not about systems randomly freezing. -
Just find it a bit concerning that Network Optix, as the engineers, don't have their ears to the ground enough to be aware of such concerning issue. The link I sent was posted at the end of April and I sent my original request the second week of May. And yes, the article references a Blackjack unit, but that unit is experiencing the exact issue I asked about in May and is running the same versions of Ubuntu and DW.
Thank you for your very dismissive response. Let's me know that this is no longer an avenue I can travel for answers as it once was when Sergey Yuldashev was responding to my concerns. -
I'm sorry to read you don't think you'll find answers here any more.
I did some research with various search queries through our ticket system back to Jan 1st 2019 and found one report that indicates that an Ubuntu system freezes, but the customer didn't provide the required information and after a few reminders we had to close the ticket.
In order to provide answers, first the problem must be clear. You indicate that your system randomly freezes, while the article refers to boot problems. Fsck is just the tool for Unix based systems that manually repairs the filesystem if possible, but says nothing about the cause. You state you use Nx Witness, the article refers specifically to the Blackjack NVR with DW Spectrum pre-loaded.
So please clarify if you experience issues with random freezing of the system or issues with booting of the system.
Also, please share if you use Nx Witness or if you use DW Spectrum?
And we would like to know which version and build you are using.Given the arbitrariness of the problem, it could be an SSD or HDD issue, but the fact that you are talking about multiple systems can suddenly be a lot smaller.
As mentioned before, in order to find the cause, I encourage you to first check the OS logs and hardware logs and if any abnormalities are found, we can decide on the next step.
Please sign in to leave a comment.
Comments
5 comments