NxMeta 4.1 Beta on Docker
Answered
Hi, thank you for signing me up to the Beta program.
I am testing NxMeta 4.1 to see how the new `additionalLocalFsTypes` setting works.
See here: https://github.com/ptr727/NxMeta-LSIO
A few questions:
1) Where is the `additionalLocalFsTypes` setting stored, I could not find it in /etc/mediaserver.conf? I want to pre-set the values for Unraid and Docker for Windows during container creation.
2) Some bugs I noticed in 4.0 is still not fixed.
2.1) `VMS_DIR=$(dirname $(dirname "${BASH_SOURCE[0]}"))` will result in the wrong directory when called from random locations.
2.2) The DEB installed is missing dependencies that are used but not referenced, `gdb gdbserver binutils lsb-release`. I expect the install to succeed when using `--no-install-recommends`.
Thanks
-
Hello Pieter,
additionalLocalFsTypes parameter is accessible via server web interface on the advanced settings page.
Here is the direct link to that page.http://localhost:7001/static/index.html#/advanced
See a screenshot.
I'm not sure I understood your point in 2.1).
Could you elaborate a bit? How and where exactly are you trying to call that command? From random locations of what in particular?
JIRA-VMS-18835
-
1) I understand how to set `additionalLocalFsTypes` from the web, I want to know where it is stored on disk. I want to set the values on the filesystem settings when I create the docker container. I don't want to have to open the UI to set values required for docker to work on my systems every time.
I'd like to set the `additionalLocalFsTypes` value in mediaserver.conf, or wherever the config is stored.
2.1) In the file /bin/mediaserver.
VMS_DIR=$(dirname $(dirname "${BASH_SOURCE[0]}"))EXE="$VMS_DIR/bin/mediaserver-bin"The calculation of VMS_DIR is wrong when called from within the same directory as the script.The BASH_SOURCE[0] value will be "." for a relative path.See several articles on stackoverflow and the BASH FAQ on how to reliably calculate the path of a script.E.g. I made this modification to illustrate:VMS_DIR=$(dirname $(dirname "${BASH_SOURCE[0]}"))
echo BASH_SOURCE[0] = ${BASH_SOURCE[0]}
echo VMS_DIR = $VMS_DIREXE="$VMS_DIR/bin/mediaserver-bin"
ARGS="-e"
echo EXE = $EXEroot@fb7ac2071669:/opt/networkoptix-metavms/mediaserver/bin# ./mediaserver
BASH_SOURCE[0] = ./mediaserver
VMS_DIR = .
EXE = ./bin/mediaserver-bin--- Starting: ./bin/mediaserver-bin -e --crash-directory=./var/crash
start-stop-daemon: unable to stat ./bin/./bin/mediaserver-bin (No such file or directory)Note that the EXE should have been mediaseserver-bin as we are already in the bin directory.
The same script hardcodes the /opt/company/mediaserver ignoring the script location, why bother calculating the script location then if it is always assumed to be in /opt/$COMPANY_NAME/mediaserver/bin?
E.g. INSTALLATION_INFO_JSON="/opt/$COMPANY_NAME/installation_info.json"
-
Hello Pieter,
You can change "additionalLocalFsTypes" setting via API /api/systemSettings.
For example, this will make zfs and xfs as additional fs.
http://admin:<your passsword>@localhost:7001/api/systemSettings?additionalLocalFsTypes=zfs,xfs
Thank you for thorough explanation about VMS_DIR
-
3) It appears as if the `additionalLocalFsTypes` only applies to primary storage, when I map multiple volumes, and want to use one for storage and one for backup, only one volume is used.
cat /proc/mounts
shfs /config fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other 0 0
shfs /media fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other 0 0
shfs /archive fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other 0 0additionalLocalFsTypes = `fuse.grpcfuse, fuse.shfs`
Storage locations only show "/media/Nx MetaVMS Media Info"
# stat /config
File: /config
Size: 12 Blocks: 0 IO Block: 4096 directory
Device: 2ah/42d Inode: 11258999079113518 Links: 1
Access: (0777/drwxrwxrwx) Uid: ( 99/networkoptix-metavms) Gid: ( 100/ users)
Access: 2020-05-18 08:12:55.430367566 -0700
Modify: 2020-05-18 08:35:28.859536001 -0700
Change: 2020-05-18 09:49:39.988700528 -0700
Birth: -
# stat /media
File: /media
Size: 32 Blocks: 0 IO Block: 4096 directory
Device: 2ah/42d Inode: 11258999079115859 Links: 1
Access: (0777/drwxrwxrwx) Uid: ( 99/networkoptix-metavms) Gid: ( 100/ users)
Access: 2020-05-18 08:12:55.430367566 -0700
Modify: 2020-05-18 08:39:01.966617843 -0700
Change: 2020-05-18 09:24:28.213499788 -0700
Birth: -
# stat /archive
File: /archive
Size: 6 Blocks: 0 IO Block: 4096 directory
Device: 2ah/42d Inode: 649362776407371830 Links: 1
Access: (0777/drwxrwxrwx) Uid: ( 99/networkoptix-metavms) Gid: ( 100/ users)
Access: 2020-05-18 08:12:55.431367548 -0700
Modify: 2020-05-18 08:12:55.431367548 -0700
Change: 2020-05-18 09:00:01.520460115 -0700
Birth: -
# -
"You can change "additionalLocalFsTypes" setting via API /api/systemSettings."
Please implement the ability to set options at provisioning time, e.g. in the config file.
Imagine you are a customer, and every time you install, you need to change configuration via an API to get the install working, pretty frustrating.
-
I think the problem with mapped volumes not showing up is that they are incorrectly detected as duplicates.
logLevel=DEBUG2
2020-05-18 10:13:55.964 422 VERBOSE nx::vms::server::fs: shfs /config fuse.shfs - added
2020-05-18 10:13:55.964 422 VERBOSE nx::vms::server::fs: shfs /media fuse.shfs - added
2020-05-18 10:13:55.964 422 VERBOSE nx::vms::server::fs: shfs /archive fuse.shfs - duplicate/archive is not a duplicate.
-
Pieter,
Could you give more detail regarding this, pleas?
2.2) The DEB installed is missing dependencies that are used but not referenced, `gdb gdbserver binutils lsb-release`. I expect the install to succeed when using `--no-install-recommends`.
DEB of which VMS version are you trying to install? What is OS specific name?
How do you determine particular DEB package depends on those?
Could give step-by-step instruction to reproduce the issue?
-
Build a container from a basic linux image, e.g. "lsiobase/ubuntu:bionic"
Specify --no-install-recommends when installing your DEB package.
Try to run, you may get various errors because the mediaserver makes use of packages that are not specified in the DEBIAN/control file.
DEBIAN/control:
Depends: zlib1g (>= 1:1.1.4), debconf (>= 0.5) | debconf-2.0, psmisc, systemd (>= 229), cifs-utils, net-tools, libcap2-bin, libldap-2.4-2, libxrender1, libgl1-mesa-glx, libxfixes3, libfontconfig1, libsm6, libice6, libaudio2, libogg0
The service fails due to a missing libgthread, probably due to missing glib in depends:
/opt/networkoptix-metavms/mediaserver/bin/root-tool-bin: error while loading shared libraries: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
GDB is not available on the host.shell_utils.sh uses GDB, but GDB is not in depends:
if ! which gdb
then
echo "GDB is not available on the host." >&2
return 0
fiThe nx reference dockerfile calls out that dependencies are missing and need to be manually installed:I believe that this may be out of date as e.g. systemd is referenced in the Depends section, and I'm not sure where bin-utils or lsb-release is used?# We need:
# - bin-utils for patching cloud host
# - lsb-release is used as a part of install scripts inside the deb package.
RUN apt-get update && apt-get install -y dbus systemd lsb-release binutils \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*Using `--no-install-recommends` is an industry good practice for creating small containers that only contain what is used.This does rely on the DEB author to completely specify required dependencies.Here is an example: https://github.com/ptr727/NxMeta-LSIO/tree/master/MinimalAny feedback on the volumes being incorrectly filtered out?
-
Hello Pieter,
Here are some news.
Regarding your provided Dockerfile for Minimal image.
We released a new MetaVMS build, which includes a fix for DEB package dependencies to make server work, when package is installed with --no-install-recommends options.
I tried it with your Dockerfile and found out there are several commands missing.
Here is what I've modified to make it be built correctly. Maybe, you should patch your copy, too.
# Install dependencies
RUN apt-get update \
&& apt-get install --yes \
# Install wget so we can download the installer
wget \
# Install nano and mc for making navigating the container easier
nano mc \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Use a local copy instead of download, else we need to install dependencies to do a download
# Copy the DEB installer file, extract it, and make a copy of the directory
RUN wget -nv -O ./vms_server_orig.deb ${DOWNLOAD_URL}
#COPY metavms-server-4.1.0.31036-linux64.deb ./vms_server_orig.debIt works ok now. I'll put a link here when the build will be published.
Build 4.1.0.31036 R7
https://meta.nxvms.com/downloads/patches
Regarding "duplicate" issue.
I've reproduced and am still in discussion with our dev. Will update once solution is found.
-
Hi Andrey
Thank you for the update.
I noticed you have a new project with docker updates on github created by Kyle.
I see that systemd is no longer required, I will try it out in my project and see if I can simplify the LSIO version.
Please ask Kyle to look at the Dockerfile, when using RUN and apt, all the commands must be in the same RUN statement.
From: https://github.com/networkoptix/nx_open_integrations/blob/master/docker/Dockerfile
RUN apt-get update -qq
# Install the mediaserver
RUN apt-get install -y /debs/${mediaserver_deb}Won't work, as each RUN is done on a separate layer, so the update will not be seen by the install.
This is the typical pattern to update, install, and cleanup in one RUN statement:
# Install the mediaserver
RUN apt-get update \
&& apt-get install --yes \
./vms_server.deb \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*If you enable issues on GitHub I can comment and provide pull requests right there ;)
-
Hi Pieter,
The Server parses /proc/mounts and in purpose does not accept several mount points to one physical partition as the separate video storages.
For example, in my setup I have a btrfs partitions and several directories there.
ls -1 /mnt/btrfs/
archive
config
media
media_b1
media_b2My docker-compose.yml:
volumes:
- /mnt/btrfs/config:/config
- /mnt/btrfs/media_b1:/media1
- /mnt/btrfs/media_b2:/media2
- /mnt/btrfs/archive:/archiveActual docker container mounts in /proc/mounts :
/dev/sda1 /media ext4 rw,relatime,errors=remount-ro 0 0
/dev/sdb8 /media1 btrfs rw,relatime,space_cache,subvolid=5,subvol=/media_b1 0 0
/dev/sdb8 /archive btrfs rw,relatime,space_cache,subvolid=5,subvol=/archive 0 0
/dev/sdb8 /config btrfs rw,relatime,space_cache,subvolid=5,subvol=/config 0 0
/dev/sdb8 /media2 btrfs rw,relatime,space_cache,subvolid=5,subvol=/media_b2 0 0As you see, /media1 and /media2 utilize the same /dev/sdb8. The Server accepts /media1 only and abandons the rest, notifying by these messages:
2020-06-02 17:29:03.736 477 VERBOSE nx::vms::server::fs: /dev/sdb8 /archive btrfs - duplicate
2020-06-02 17:29:03.736 477 VERBOSE nx::vms::server::fs: /dev/sdb8 /config btrfs - duplicate
2020-06-02 17:29:03.736 477 VERBOSE nx::vms::server::fs: /dev/sdb8 /media2 btrfs - duplicate -
I understand, but that logic does not work on Unraid, where the behavior of the mount backing IO is dictated by the Unraid FUSE code, and your code only sees FUSE, not what it does behind the scenes.
For example the /archive mount can point to spinning disks for long term write once type storage, while the /media mount can point to a BTRFS SSD array for fast read and writes, and those same files are later moved to spinning disks when the SSD runs low on space.
The mount volume your code sees does not know if the files are currently on SSD or on spinning disks, it just sees the FUSE attributes, and it can't distinguish the very different behaviors.
So this leaves few options; I can either only mount one volume on Unraid, or you could add an option to turn off duplicate filters, or you can let me specify storage locations in the config file, and forego the auto detect logic?
As for configuring filesystems at provisioning time, I did some debugging and I see you store the values in the SQLite DB file.
The same DB file contains all sorts of other config data, so it would require code to dynamically insert the value before the service starts, complex.
How about the ability to set the options in the config file, and at runtime you can copy the settings to the DB file, this way I can set the config file at provisioning time?
-
I updated my project and removed the systemd modifications, also added --no-install-recommends, seems to work, thank you, so much easier to maintain now that I don't have to modify the DEB file at install time.
See: https://github.com/ptr727/NxMeta-LSIO
-
FYI, I converted my DWSpectrum project to the simplified non-systemd version, and other than losing my license keys (cloud licensing please), it seems to work fine.
See: https://github.com/ptr727/DWSpectrum-LSIO
-
After my DWSpectrum licenses became invalid, and DW is giving me a hard time to fix that, I started looking into why the hardware id changed.
I can see that the hardware identifiers that are detected changed when I changed the container to be based on the new simplified build.
Notice how the detected attributes changed.
2020-06-03 15:24:48.615 332 ALWAYS START: ================================================================================
2020-06-03 15:24:48.616 332 ALWAYS START: Digital Watchdog Media Server started, version: 4.0.0.30917, revision: 11f453e4a7bb
2020-06-03 15:24:48.616 332 ALWAYS START: Binary path: /opt/digitalwatchdog/mediaserver/bin/mediaserver-bin
2020-06-03 15:24:48.616 332 ALWAYS START: Log level: info
2020-06-03 15:24:48.616 332 ALWAYS START: Log file size: 10M, backup count: 25, file: /opt/digitalwatchdog/mediaserver/var/log/hw_log.log
2020-06-03 15:24:48.616 332 ALWAYS START: Mutex implementation: qt
2020-06-03 15:24:48.618 332 INFO HWID: {"biosID":"0123456789","biosManufacturer":"American Megatrends Inc.","boardID":"ZM137S033113","boardManufacturer":"Supermicro","boardProduct":"X10SLH-F/X10SLM+-F","boardUUID":"00000000-0000-0000-0000-002590d4c2e6","compatibilityBoardUUID":"00000000-0000-0000-0000-002590d4c2e6","date":"2020-06-03T15:24:48","mac":"00:00:00:00","memoryPartNumber":"","memorySerialNumber":"","nics":[{"mac":"00:25:90:D4:C2:E7","xclass":"XXX"},{"mac":"02:42:B5:D7:83:4F","xclass":"XXX"},{"mac":"00:25:90:D4:C2:E6","xclass":"PCI"},{"mac":"00:25:90:D4:C2:E7","xclass":"PCI"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"CE:89:91:29:D8:8A","xclass":"XXX"}]}
2020-06-03 15:24:48.619 332 INFO HWID: Hardware IDs: ["74be16979710d4c4e7c6647856088456", "0112f516cd7b3177d9a7399e20d85f12e0", "0212f516cd7b3177d9a7399e20d85f12e0", "0312f516cd7b3177d9a7399e20d85f12e0", "04ec72b508a5e3b15af7e8a2e940d14f59", "04ec72b508a5e3b15af7e8a2e940d14f59", "04ec72b508a5e3b15af7e8a2e940d14f59", "04ec72b508a5e3b15af7e8a2e940d14f59", "0478b605f96c18bf39989bc0a4a7ddf1ba", "0421683e61b7bb2fbc76418d93f69cfbaa", "0421683e61b7bb2fbc76418d93f69cfbaa", "048662144a4d9bd3716ebc786535b132c6", "04398e72a5947f67a2337afe9f005fab2f", "04d41d8cd98f00b204e9800998ecf8427e", "05ec72b508a5e3b15af7e8a2e940d14f59", "05ec72b508a5e3b15af7e8a2e940d14f59", "05ec72b508a5e3b15af7e8a2e940d14f59", "05ec72b508a5e3b15af7e8a2e940d14f59", "0578b605f96c18bf39989bc0a4a7ddf1ba", "0521683e61b7bb2fbc76418d93f69cfbaa", "0521683e61b7bb2fbc76418d93f69cfbaa", "058662144a4d9bd3716ebc786535b132c6", "05398e72a5947f67a2337afe9f005fab2f", "05d41d8cd98f00b204e9800998ecf8427e"]
2020-06-03 15:25:21.665 315 ALWAYS START: ================================================================================
2020-06-03 15:25:21.665 315 ALWAYS START: Digital Watchdog Media Server started, version: 4.0.0.30917, revision: 11f453e4a7bb
2020-06-03 15:25:21.666 315 ALWAYS START: Binary path: /opt/digitalwatchdog/mediaserver/bin/mediaserver-bin
2020-06-03 15:25:21.666 315 ALWAYS START: Log level: info
2020-06-03 15:25:21.666 315 ALWAYS START: Log file size: 10M, backup count: 25, file: /opt/digitalwatchdog/mediaserver/var/log/hw_log.log
2020-06-03 15:25:21.666 315 ALWAYS START: Mutex implementation: qt
2020-06-03 15:25:21.670 315 INFO HWID: {"biosID":"","biosManufacturer":"American Megatrends Inc.","boardID":"","boardManufacturer":"Supermicro","boardProduct":"X10SLH-F/X10SLM+-F","boardUUID":"","compatibilityBoardUUID":"","date":"2020-06-03T15:25:21","mac":"00:00:00:00","memoryPartNumber":"","memorySerialNumber":"","nics":[{"mac":"00:25:90:D4:C2:E7","xclass":"XXX"},{"mac":"02:42:B5:D7:83:4F","xclass":"XXX"},{"mac":"00:25:90:D4:C2:E6","xclass":"PCI"},{"mac":"00:25:90:D4:C2:E7","xclass":"PCI"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"00:00:00:00","xclass":"XXX"},{"mac":"CE:89:91:29:D8:8A","xclass":"XXX"}]}
2020-06-03 15:25:21.670 315 INFO HWID: Hardware IDs: ["74be16979710d4c4e7c6647856088456", "01d41d8cd98f00b204e9800998ecf8427e", "02d41d8cd98f00b204e9800998ecf8427e", "03d41d8cd98f00b204e9800998ecf8427e", "04dbb9e1189359811f2a150b848e1c629b", "04dbb9e1189359811f2a150b848e1c629b", "04dbb9e1189359811f2a150b848e1c629b", "04dbb9e1189359811f2a150b848e1c629b", "043abc4706a6be75d685bb00e0a19bad22", "0432c8432a0207be098bccbf8dd12d3611", "0432c8432a0207be098bccbf8dd12d3611", "0477175710d0e6803553c2ba1ba5f69ade", "0409c3ae106b347c57e1e356c3756a7748", "04d41d8cd98f00b204e9800998ecf8427e", "05dbb9e1189359811f2a150b848e1c629b", "05dbb9e1189359811f2a150b848e1c629b", "05dbb9e1189359811f2a150b848e1c629b", "05dbb9e1189359811f2a150b848e1c629b", "053abc4706a6be75d685bb00e0a19bad22", "0532c8432a0207be098bccbf8dd12d3611", "0532c8432a0207be098bccbf8dd12d3611", "0577175710d0e6803553c2ba1ba5f69ade", "0509c3ae106b347c57e1e356c3756a7748", "05d41d8cd98f00b204e9800998ecf8427e"]"biosID":"0123456789" vs. "biosID":""
"boardID":"ZM137S033113" vs. "boardID":""
"boardUUID":"00000000-0000-0000-0000-002590d4c2e6" vs. "boardUUID":""
"compatibilityBoardUUID":"00000000-0000-0000-0000-002590d4c2e6" vs. "compatibilityBoardUUID":""And then the computed hashes changed, invalidating my license.
I don't know how you are computing the DMI information, as tools like dmidecode does not work as /dev/mem is not accessible from docker.
But, /sys/class/dmi/id/ is available, and you can get all the equivalent information without privilidged mode.
cat /sys/class/dmi/id/board_serial
ZM137S033113
# cat /sys/class/dmi/id/product_serial
0123456789
# cat /sys/class/dmi/id/product_uuid
00000000-0000-0000-0000-002590d4c2e6Given the draconian behavior of DW when it comes to licensing (3 changes and you have to buy a new license), the changes in HWINFO on the same hardware will cause ongoing problems.
Please advise?
-
Hi Pieter,
To make licenses persistent you could setup a dedicated server either virtual or bare-metal and make it not to change its hardware configuration. In case of virtual make it not to migrate to any other host. Connect this server to your VMS system. Regain for its HW ID and reapply them. This server will serve only for keeping licenses bound to its HW ID. In this case, your virtual servers in containers can migrate and change their HD ID without lousing licenses, since the licenses are always available in the System.
-
Pieter, is it helps, we give out licenses for testing and development for free (using Nx Meta): https://support.networkoptix.com/hc/en-us/articles/360045718294-Getting-Licenses-for-Developers
Meanwhile, we understand the problem, and we want to change license logic to be system-specific instead of hardware-specific, it is on the roadmap, but I don't know when will we be able to get to it. A question regarding that: are you with connecting your system to the cloud?
-
Thanks, I did get a NxMeta test license.
Sorry, I think I was not specific enough, let me try again:
The mediaserver detects hardware information, and uses it for licensing.
As I made changes to how the container is built, following your new method that does nto rely in systemd, the attributes that were collected by the same version of code, on the same hardware, changed, and invalidated my license.
Since the hardware did not change, my license should not have changed.
I showed you an example on how to detect the missing attributes, with a method that is reliable on docker.
I ask that your code be fixed to reliably detect the attributes.
Yes, I hope you move away from hardware tied licenses, but this is a bug, not a feature request.
-
So can you successfully use btrfs as a volume with Nx ?
I added it as per https://support.networkoptix.com/hc/en-us/community/posts/360044241693/comments/360011590433
but it still came up with a 'incompatible volume' error in the server settings after a restart...
-
Currently, we only support ext4 officially, others are experimental, but we will keep it in mind for future releases.
-
Pieter Viljoen apologies for the novice question, but does running the container with the host network setting give it a consistent MAC address and keep the license valid between updates ?
Please sign in to leave a comment.
Comments
30 comments