Storage selection on Ubuntu and the 'reserved' status
Completed
Once again difficulties with upgrading from 3.2 to any later version, in this case 4.2, and the management and use of storage.
Configured storage meets the 10% rule - local OS storage is much less than 10% of the largest drive of 10Tb
Currently no recording is happening. This can be seen by several obvious indicators...
* The red recording dots beside each camera blink off in rolling camera succession about once per second
* The timeline bar scrolls through time continuously, because it has nothing to show.
2022-05-11 08:32:00.760 2926 INFO QnRecordingManager(0x7fb5ec09d340): Recording is stopped for camera 58-
All the storage locations are greyed out in the server storage management page.
Why do storage locations become greyed out, and hence un-editable?
Advanced additionalLocalFsTypes is set to ext4
What does "Reserved" beside a storage location mean? Why can this storage not be used for recording?
Also seeing this on startup
2022-05-11 13:42:50.348 15949 WARNING QnFileStorageResource(0x7f8348003200, d193301e-79a3-60f4-295f-6a3de1c4ef12): canWrite: Open file '/srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e.tmp' (direct access) for writing failed. Error: Failed to remove previous file
2022-05-11 13:42:50.348 15949 INFO QnStorageDb(0x7f83c0004890): Loading chunks from DB. storage: /srv2/HD Witness Media, file: /srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e_media.nxdb
2022-05-11 13:42:50.368 15995 WARNING nx::vms::server::analytics::AnalyticsDb(0x7f834800b730): Unable to give read access to the mount point: /srv2/HD Witness Media/..
2022-05-11 13:42:50.458 15946 WARNING QnStorageDb(0x7f83c0004890): Failed to parse DB file /srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e_media.nxdb
2022-05-11 13:42:50.468 15946 WARNING QnStorageDb(0x7f83c0004890): Loading chunks from DB failed. storage: /srv2/HD Witness Media, file: /srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e_media.nxdb
Advanced server settings
Storage
| URL | Free Space | Reserved Space | ||
|---|---|---|---|---|
| /srv2/HD Witness Media |
28.23 GB of 10.66 TB
|
GB
|
-
Template data, for completeness
Summary: NX will not record video, because it cannot write to disk, because of an apparent permissions issue.
Environment:
Nx Witness Version: 4.2.0.32840
Client OS: 4.2.0.32840
Server: 4.2.0.32840
Network Topology: Servers 1, cameras 6, clients 3, and switches 2
Special features: Server virtualized on ProxmoxReproduction Scenario: Install 3.2, works correctly. Storage on a second local ext4 drive. Then install any of 4.0, 4.1, 4.2, they all exhibit the issue. Recording stops working.
Attachments: See above for scenarios and content
-
Hi David McNeill,
That a lot of scattered text. Next time, please consider using the Support Request Template in THIS support article for the entire request, and share logs as downloadable links.
The greyed out storage mean that there is storage, but Nx can't write to it. The logs below you share confirm this:
2022-05-11 13:42:50.348 15949 WARNING QnFileStorageResource(0x7f8348003200, d193301e-79a3-60f4-295f-6a3de1c4ef12): canWrite: Open file '/srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e.tmp' (direct access) for writing failed. Error: Failed to remove previous file
2022-05-11 13:42:50.348 15949 INFO QnStorageDb(0x7f83c0004890): Loading chunks from DB. storage: /srv2/HD Witness Media, file: /srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e_media.nxdb
2022-05-11 13:42:50.368 15995 WARNING nx::vms::server::analytics::AnalyticsDb(0x7f834800b730): Unable to give read access to the mount point: /srv2/HD Witness Media/..
2022-05-11 13:42:50.458 15946 WARNING QnStorageDb(0x7f83c0004890): Failed to parse DB file /srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e_media.nxdb
2022-05-11 13:42:50.468 15946 WARNING QnStorageDb(0x7f83c0004890): Loading chunks from DB failed. storage: /srv2/HD Witness Media, file: /srv2/HD Witness Media/eb0f1108-ab6b-fc46-9108-2fc9d9ffef6e_media.nxdbPlease check THIS article, to change the storage permissions.
For the analytics storage issue, please refer to THIS support article. -
Thank you for your reply. I am well aware of how to change linux permissions. The issue is what they should be set to? Is there any NX document anywhere that says what they should be? It would seem that the media server process running as the user networkoptix...
network+ 24501 1 0 May11 ? 07:38:48 /opt/networkoptix/mediaserver/bin/mediaserver-bin -e --crash-directory=/opt/networkoptix/mediaserver/var/crashwould have no trouble writing to a directory that it owns...
drwxr-xr-x 2 networkoptix networkoptixIt could also be that there is some other issue, that is surfacing as a write issue, but perhaps has a different cause.
Linked is the current log file from startup, showing the only enabled camera can't get started recording. Although there are errors in the log file, it is not apparent how to resolve them, or what the root cause is, perhaps you can review and suggest. The storage reindexing is currently happening every startup, for unknown reasons.
-
2022-05-12 11:03:27.280 12273 ERROR QnFileStorageResource(0x7fb7100032c0, d193301e-79a3-60f4-295f-6a3de1c4ef12): Rename '/srv2/HD Witness Media/low_quality/58-50-ED-14-A3-4D/2022/05/12/11/1652310
132157.mkv' to '/srv2/HD Witness Media/low_quality/58-50-ED-14-A3-4D/2022/05/12/11/1652310132157_74941.mkv' failed
2022-05-12 11:03:29.013 12359 ERROR QnFileStorageResource(0x7fb7100032c0, d193301e-79a3-60f4-295f-6a3de1c4ef12): Rename '/srv2/HD Witness Media/low_quality/58-03-FB-81-BB-35/2022/05/12/11/1652310
133849.mkv' to '/srv2/HD Witness Media/low_quality/58-03-FB-81-BB-35/2022/05/12/11/1652310133849_74980.mkv' failed -
Storage access and write test passes
2022-05-12 11:08:45.410 2780 DEBUG QnStorageManager(0x7fb77c05b240, Normal): Starting to test storages
2022-05-12 11:08:45.410 2780 DEBUG QnStorageManager(0x7fb77c05b240, Normal): Testing storage /srv2/HD Witness Media
2022-05-12 11:08:45.418 2780 DEBUG nx::vms::server::fs: Return { /dev/mapper/pve-vm--601--disk--1 -> / ext4 0/8320901120, /dev/mapper/vmdata3-vm--601--disk--0 -> /srv2 ext4 0/11716955471872, cgr
oup -> /sys/fs/cgroup/pids cgroup 0/0, debugfs -> /sys/kernel/debug debugfs 0/0, devpts -> /dev/pts devpts 0/0, fusectl -> /sys/fs/fuse/connections fusectl 0/0, hugetlbfs -> /dev/hugepages hugetlbfs
0/0, mqueue -> /dev/mqueue mqueue 0/0, none -> /dev tmpfs 0/503808, proc -> /proc proc 0/0, pstore -> /sys/fs/pstore pstore 0/0, securityfs -> /sys/kernel/security securityfs 0/0, sysfs -> /sys sys
fs 0/0, tmpfs -> /run tmpfs 0/12630159360 }
2022-05-12 11:08:45.418 2780 DEBUG QnFileStorageResource(0x7fb7100032c0, d193301e-79a3-60f4-295f-6a3de1c4ef12): [initOrUpdate] IsMounted check succeded for '/srv2/HD Witness Media', local path:
'/srv2'
2022-05-12 11:08:45.418 2780 DEBUG QnFileStorageResource(0x7fb7100032c0, d193301e-79a3-60f4-295f-6a3de1c4ef12): canWrite: Write test via root-tool succeeded
2022-05-12 11:08:45.419 2780 DEBUG QnStorageManager(0x7fb77c05b240, Normal): Testing storage /srv2/HD Witness Media done -
Once the file reindex finishes (about 4 hours), the storage changes from status Main with trying and failing to write or rename mkv files to Reserved, and no longer even trying to write files.
I've tried changing the directory tree to owned by root, as this is how other NX systems configure themselves, but it doesn't make any difference.
So information on the mkv file write logic process, and the julian date directory creation process would be very helpful.
-
Hi David McNeill,
Regarding:
Once the file reindex finishes (about 4 hours), the storage changes from status Main with trying and failing to write or rename mkv files to Reserved, and no longer even trying to write files.
This typically happens when the disk if full of data, that isn't managed by Nx. The drive needs to have at least >50 GB of free space and has to be 10x bigger than the OS drive, to become available for storage and the before mentioned clarifies why no files are attempted to be written to the storage.
-
So that's easily fixed, delete some old video to bring the drive back to more than 50Gb available, which I did.
NX used it briefly for a few minutes, and designated it main, then failed, with unable to create file, same issue.
Something else is going on here. What do you suggest for debugging it?
-
Hi David McNeill,
I'll convert this topic into a support ticket and we'll proceed there. -
Thanks. Can you email me the link please. It hasn't appeared in any consoles.
David
-
Hi David McNeill,
No sure if this works, but THIS is the link.
-
oops
The page you were looking for doesn't existYou may have mistyped the address or the page may have moved
Take me back to the home page -
Hi David McNeill,
Most likely this was caused, since the link was only intended for agents, not for users.
I tried something else, can you check if the ticket appears in your console overview?
-
The problem turned out to be shallow mounts, like /srv
Storage media folders near to root like '/srv/HD Witness Media' do not work correctly in versions after 3.2
The new root-tool that does the directory creation and transit file renaming fails with permission errors, even though it is not a permission issue. It just refuses to work with mounts directly off root.
Storage media mounts should be down a level, for example /mnt/camera-storage-1/
This applies to all linux platforms including physical and virtual.
Post is closed for comments.
Comments
17 comments