Two merged servers sharing same storage volume?
Answered
Ideas for MMJ installation. Thinking outside “the box”.
- Currently 50 cameras and wanting 20 more.
- Very remote location
Will use two merged servers. Should one server have an issue that cannot be resolved locally or with TV, it can be UPS’d back to me in a day. This will leave the surviving server to handle the entire load for a few days.
But here is the question.. What happens if a server goes down? Well, the cams on the failed server fail-over to the surviving server and continue to record. However, the historical video is unavailable.
Adding a NAS where both servers record to sounds reasonable. But, because it can be problematic for both servers to record to the same volume (at least it used to be pre 4.0). If I could connect both NAS volumes to both servers but make one of them “read only”, that might work but I can’t designate primary or backup storage as read only.
The rub here is that the state can and does make unannounced inspections and expects to be able to see 40 days of history. If they happen to show up when a server is down, we have some explaining to do.
What can you recommend? My thoughts are…
- Make it so two servers can read/write to the same volume (as I recall, the problem was the index file, not so much the video files)
- Make it so I can set a volume as read only so server A will only write to volume A but can read from volume B.
- Option I don’t know about yet (where you come in)
-
Same volume but different folders? Hmmm, I will have to think on that.
I assume you mean N:\nvr1\HD Witness Media\... and N:\nvr2\HD Witness Media\..?
But will this not create the same issue where two servers are writing two different index files to the same "folder"? If I want NVR2 to be able to read from the NVR1 folder, I will have to include it as a storage location correct? Will that not also cause NVR2 to write to the NVR1 folder as well?
Did I ask the question correctly am I just being obtuse (I am hoping for obtuse)? Two NVR's and a NAS. NVR1 fails. How can NVR2 read the video archive for NVR1's folder (from before NVR1's failure and after all the cameras have failed over to NVR2)? That actually is simple...add both folders as storage locations. But more importantly, how do I prevent NVR2 from writing to NVR1's folder when NVR1 is restored? Then I would have two servers writing two index files to the same folder.
If I were to intentionally take NVR1 offline, I could enable NVR1's folder as storage in NVR2 and reverse the process when I restored it. But that is not the scenario I am dealing with. I think It could work if I could make NVR1's folder RO in NVR2's FSTAB, but that won't work. Read Only storage shows "inaccessible" -
I'm interested in this as I've been looking at similar scenarios.
Set each machine as failover. Install a NAS. Record to internal drives in each machine, with real time back up to the NAS.
Thinking this would work?
Each machine records- tick
One machine fails, other picks up its cameras- tick
Failed machines pre-fail footage is locatable on the NAS- tick
If I am thinking down the right track, Nx will know where to find it.
-
I have done exactly that (almost) in the past but have suffered some data anomalies. Usually rebuilding the backup index will solve the problem but not always entirely. What I did different was to do a scheduled daily backup to the NAS beginning at 6PM when the plant shut down. Reason vs. real-time was "If a real-time backup cannot be executed because of an unexpected failure (e.g. Backup NAS not running or network connection is dropped) the backup will be lost for this time period. Once the NAS or network issue is fixed the backup will continue backing up in real-time from the point at which the failure has been rectified" The issue (I believe) is that there should only be one server writing to the index file at a time, hence Derek's recommendation of separate folders, separate index files. Probably the same can be said for the obj_detection db file(which can now be located on one of the internal drives but that defeats the purpose in this case). I am going to recreate this again in my shop and do a better job of logging. The index file should be named for the HWID so there so there should be two index files right? No conflict. The video mkv's should not conflict because they would be time specific. Yes, it SHOULD work.. I'll know more about why it has problems when I get it set up and pay more attention. I just would like to understand it better.
-
You are correct about the archive index per server, and if they were to all be to one location, issues could arise with lost footage, etc. as they all share the same location.
Glynn, the same would apply to the servers sharing the NAS as their backup location
For using the same NAS for multiple servers (assuming they are folder organized), you should be aware of the precautions:
- If failed server recordings are to be used, you would need to make sure the failed server archive data is migrated to the working archive folder then "Reindex Archive" is performed.
- The Reserved storage should be increased (try double the value) in the advanced settings of each of the two servers, as a precaution for NAS (especially RAID setup) drive not running out of space, which could lead to data loss problems.
- Be mindful that the read/write speed will be a greater demand for the NAS drive if multiple servers are using it.
Other than that, if you configure two separate paths to the folders on the NAS drive, it should work and prevent any archive loss so long as you take precautions.
Please let me know if that answers both of your questions.
-
One final thought... Using folders as suggested, when adding the folder as a storage location, the space available will show the space for the entire drive (naturally). So, with two NVR's writing to two folders on the same drive, will the system behave properly when the drive gets full? Example NVR1 to Folder1 is writing twice as much data as NVR2 is to Folder2. When the drive gets full, how will that be handled? Deletion of oldest data in the drive regardless of folder, or oldest in the folder with the index file? Just curious what to expect.
-
Nope... still don't get it..
Two merged servers backing up to two separate volumes or folders on a volume (as suggested by TS). Now, Server1 fails and its cams fail over to the surviving Server2… check. But for Server2 to read the archive from Server1’s backup, Server1’s backup folder must be added as a storage location to Server2. Now what happens? Server2 starts writing it backups to Server1’s folder. Now we have the problem… a volume/folder with two index files, which as I understand from TS, is not recommended.
When adding a storage location, if Nx cannot write to it, it does not recognize it. So simply adding RO in the FSTAB won’t work.
Would the ability to add storage as RO solve the problem of multiple index files in the same volume/folder? Or is there another solution?
Is this the only way?..."- If failed server recordings are to be used, you would need to make sure the failed server archive data is migrated to the working archive folder then "Reindex Archive" is performed."
That seems like a workaround to a problem that should not exist and was not considered in the engineering.. I sense a Feature Request coming..
-
Hey Paul,
Did you get anywhere further with this?
I've just been ever increasing the allocation to our NAS volume to get around the 'volume is full' problem, however i cannot do this much more due to other volume allocations.(3x Windows Servers as a cluster with a single NAS volume - a folder per Server)
Thanks,
Please sign in to leave a comment.
Comments
10 comments