HDD too slow warnings, then database corruption and crash
Hi I looked at a server from a client which had inaccessible storage. There seems to be no hardware error, so eventually after investigating, the client asked me to wipe the drives and bring the server back into production asap, but I would like to prevent this problem in the future. Here's what I could find on what happened:
Nx Witness 6.0.5.41290, Ubuntu 22.04.4 LTS. 4 local SATA drives, ext4: 3x Seagate Exos X20 20TB, 1x Exos X16 10TB. 25 cameras, , continuous recording. Installed May 2024.
Aug 26 2026, 20:46:48, mediaserver process gets killed. Crash dump:
ASSERTION FAILED: .../media_server_process.cpp:1266 (false) Server was unable to stop within 100s
After restart, main.log shows this every second for 100+ sec:
WARNING nx::vms::server::StorageDb: Failed to parse DB file /data3/camerastorage/HD Witness Media/<GUID>--1.nxdb
WARNING nx::vms::server::StorageDb: startDbFile write DB header failed
WARNING nx::vms::server::StorageDb: Vacuum failed. Retrying
21:01:52, second crash, same assertion.
After both crashes 3 of 4 storage locations show 0-byte .nxdb/.tmp files, marked inaccessible/reserved in Storage Management.
SMART passed on all drives. fsck: superblock said clean, but a full e2fsck -f found and fixed filesystem errors on the affected drives.
Went through the archived main_*.log.zip files back to install date. Found this recurring entry:
WARNING nx::vms::server::ServerStorageStreamRecorder: HDD/SSD is slowing down recording for camera '{GUID}'. 1001 frames have been dropped!
WARNING QMessageLogContext: Temporary disable rebuild archive (if active) due to high disk usage slow down recorders. Always exactly 1001 frames, never anything else.
Before the rebuild all 4 drives sat at ~95% used. e2freefrag showed ~25% nominally free but largest contiguous extent only ~90MB, most free space in 8-32MB fragments.
Ran dd (2GB, direct IO) before and after wiping/reformatting the same 4 drives:
data1: 210/235 MB/s -> 258/278 MB/s
data2: 220/197 -> 271/276
data3: 214/168 -> 229/245
data4: 156/176 -> 260/271
These are all Seagate Exos Enterprise drives
Questions:
What's the actual trigger behind "HDD/SSD is slowing down... 1001 frames have been dropped"? Is 1001 a fixed buffer threshold, is a lower bitrate the only solution for this?
What does media_server_process.cpp:1266 / "Server was unable to stop within 100s" actually check? What normally causes a thread to not exit in that window?
Any recommended recovery when a .nxdb is corrupted and Archive Index Rebuild doesn't fix it, other than wiping the storage?
Any official guidance on e2freefrag/fragmentation monitoring for ext4 under continuous recording, or a recommended reserved space margin above the default?
Can post more logs if needed. The server is back up and running, but I'd like to know more about the cause so I can prevent something like this.
-
Well with spinners I'd persnoally never let it go about 85% utilization but safer at 80% Even less when in raided system like zfs but thats anoth story. The closer a drive reaches 100% files will lose more and more continous space for recordings. I know you said only about 25% fragmentation. But with continuos recording every little bit can help.
Then there's the other thing I noticed. I have all SSD storage in our systems and I have seen the same errors. Maybe not as many as yours but I even even went to the recordings at the specific time of the errors and found no missing frames when going frame by fram in vlc.
Have you attemped a DB backup followed by a DB restore to see if that resolves anything. Id attempt the DB restore on a test system first to make sure it actually restores though.
Just my two cents thrown in the bucket. Not much but just some thoughts
0
Please sign in to leave a comment.
Comments
1 comment