Darknet model integration
Answered
I’m currently working on the integration of our own AI object detector and classifier, based on darkness DLL, in your “metadata_sdk” in order to implement the plug-in. So I implemented our AI model in darkness and used it as a function inside “bool DeviceAgent::pushUncompressedVideoFrame(const IUncompressedVideoFrame* videoFrame)” that takes the video frame and converts it to darknet “image_t” then uses our AI model to detect and classify the objects.
The entire project compiles successfully!
However, when the plug-in is moving to its appropriate folder “..\..\Network Optix\Nx Witness\MediaServer\plugins” the plug-in despairs from cameras options and the objects tab as well. Here some screenshots of the plug-in and the programs.
and
I also found a similar question that was asked to your team support on NX developers’ website, but didn’t find an appropriate answer.
Could you please help us to integrate our AI model, based on Darknet, in the plug-in through “metadata_sdk”?
Eager reading from you! Have a nice day!
-
Hello Riles Ouali,
Here is the post worth reading on how to organize third library binaries in a plugin.
https://support.networkoptix.com/hc/en-us/community/posts/360045632473/comments/360011473654
Do you link you daknet library statically or dynamically? Where the .dll file of darknet is located in your system?
Could you try to do the following:
1. configure your IDE to run Server under debugger. Here is an example for Visual Studio
2. Setup a break point in the plugin code somewhere in Engine::~Engine() or new the line where you call darknet function for the first time.
3. Build the plugin and copy to server's plugin directory.
4. Run the Server under debugger in your IDE. The execution should stop at the breakpoint.
5. Step further in debugger and find out the line at which code fails.
6. Share with us the code of the function this line belongs to.
TTPLUGINLIBS
0 -
Hello Andrey Terentyev
_ I organized my third library binaries the same way as in this description, but still nothing appears in panel
_ Do you link you daknet library statically or dynamically? I tried both: statically and dynamically!
_ Where the .dll file of darknet is located in your system? In the same plug-in directory (the same folder as the plug-in .dll)
_ Could you try to do the following:
- Configure your IDE to run Server under debugger.
- Setup a break point in the plugin code somewhere in Engine::~Engine() or new the line where you call darknet function for the first time.
- Build the plugin and copy to server's plugin directory.
I followed this threes instruction, but I couldn’t run the Server under debugger in my IDE. Could you please help me to do it?
_ Share with us the code of the function this line belongs to. I sent you the code by email! Could you please let me know if you received it?
Regards!
0 -
Hello Riles Ouali,
Yes, I've got your code. Please, allow me some time for investigation.
0 -
Hello,
I've got your code archive and had a look into it.
Thought I got the code compiled successfully a plugin is not loaded with an error
2020-06-10 14:59:44.539 2c88 ERROR PluginManager(0x2b777e62340): Failed loading Server Plugin [C:/Program Files/Network Optix/Nx MetaVMS/MediaServer/plugins/stub_analytics_plugin.dll]: [cannotLoadLibrary]: Cannot load library C:\Program Files\Network Optix\Nx MetaVMS\MediaServer\plugins\stub_analytics_plugin.dll: The specified module could not be found.
The resulting dll does not export nxCreatePlugin function, which is to be invoked by the Server to initialize plugin.
You don't use Plugin helper class in your code normally exporting this function neither you implement and export nxCreatePlugin explicitly.
Please, see plugin.cpp and plugin.h in stub_analytics_plugin example.
Please, make sure you are using the latest Metadata SDK available here https://nxvms.com/downloads/patches
0 -
The good practice in developing a plugin is to start with an example plugin. stub_analytics_plugin for example.
1. Build it with the build_samples.bat scripts provided with the SDK. Place it to the server and check if the server loads it successfully.
2. Configure your IDE and try to build the same original plugin from IDE. Place it to the server and check if the server loads it successfully.
3. Change some settings values in plugin or engine or device agent manifests. Build a plugin, place it to the server and check if the server loads it successfully.
4. Make some minor changes to the code. For example, include (just include without invoking functions) your darknet library. Build a plugin, make sure your external library is linked. Place a plugin to the server and check if the server loads it successfully.
0 -
_ The resulting dll does not export nxCreatePlugin function, which is to be invoked by the Server to initialize plugin.
- How can I include or create nxCreatePlugin?
_ You don't have plugin helper class implemented in your code normally exporting this function.
- I used the same helper class that is provided by the “Media SDK”. Could you please provide me an example of its implementation or guide me to implement my own version?
_ Please, see plugin.cpp and plugin.h in stub_analytics_plugin example.
- I did! But it sounds, somehow, can’t figure it out!
_ Please, make sure you are using the latest Metadata SDK available here
- The code I sent you is the latest “Metadata SDK” that I found of your website on April first! The only thing that I changed, is including some libraries in the project configuration and creates a static AI_Detector in device_agent.h and a function that invoking that detector “processFrame” used one in pushUncompressedVideoFrame. That’s it!
- If the function “processFrame” is commented, the plugin’ll appear once again in the client panel!
- I tried to download the latest “Metadata SDK” with the link you sent me but it’s not working https://nxvms.com/downloads/patches (page not found)
_ Concerning the practice of creating a plug-in, it’s exactly what I did!
_ Could-you please provide us with more details about debugging the media_SDK when invoking as you suggested it in the following link? : https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/3600115167740 -
1. You should create an account and sign up for Meta VMS. There is a pinned thread "Getting started" on the top of "Developer forum"
2. log in to meta cloud and download the latest (at the moment 4.1.0.31036 R7) Metadata SDK from here https://meta.nxvms.com/downloads/patches
Sorry, I provided wrong link in my the last post.
You would probably want to download the latest MetaVMS build to develop your plugin for.
3. In Metadata SDK you'll find the "samples" folder which contains two example plugins.
- sample_analytics_plugin
- stub_analytics_plugin
You could use any of them to rebase your code into.
Please, let me know when you are done with rebasing.
Please, share your code again if you did not get your plugin working after rebase.
0 -
_ Could-you please provide us with more details about debugging the media_SDK when invoking as you suggested it in the following link? : https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011516774
Could you please specify what exactly remains unclear? I would focus on this.
0 -
I created and activated my account successfully. Then, when I tried to download the patches from https://meta.nxvms.com/downloads/patches I got the same error:
0 -
Hi,
Here is the link to proceed to early access program.
https://support.networkoptix.com/hc/en-us/articles/360046713714-Get-an-Nx-Meta-Build
Here is the page worth reading as well
https://support.networkoptix.com/hc/en-us/categories/360000737654-Develop-with-Nx-Meta
0 -
_ The latest version I found is (4.0.0.29991 R17)
_ Just subscribed: https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011784293
_ I would like to step trough the plugin, in debug mode, when developing it to identify and figure out any problem when integrating a third party in meta_SDKS: anwer to https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011784013
0 -
You see, there is "Download Nx Meta VMP" link at the bottom of the screen.
0 -
_ I would like to step trough the plugin, in debug mode, when developing it to identify and figure out any problem when integrating a third party in meta_SDKS: anwer to https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011784013
I don't know which IDE are you using. And frankly speaking this kind of questions is out of scope of Metadata SDK.
In Visual Studio is just like this.
- You configure IDE like it was shown on s screesnhot. https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011516774
- Build your plugin.
- Copy your plugin to the Server. (Make sure your server is stopped)
- In VS in the code of your plugin set a break point where you want.
- In VS start debugging by clicking Debug->Start debugging or by pressing F5.
- VS will start the Server and will stop execution at your break point.
Here is an article for Visual Studio for more details on navigation.
0 -
_ I'm using visual studio 2017!
_ I followed the instructions as you provided here, and automated the procedure of transferring the plugin with a bat filenet stop "Network Optix Media Server"
set CURRENT_DIR=%~dp0
move %CURRENT_DIR%stub_analytics_plugin.dll "C:\Program Files\Network Optix\Nx Witness\MediaServer\plugins"n
But when I hit F5 I got this following message
0 -
It seems VS is trying to start program with the path "F:\Plug-ins... etc"
Which leads to conclusion
_ I followed the instructions as you provided here
seems to be wrong statement.
Please, once gain look carefully at the screenshot in this post
https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011516774
In VS right click on "stub_analytics_plugin" project. Select "Properties".
In the left part of the opened window select "Debugging" and fill in "Command" and "Command Arguments" as shown at the screenshot.
0 -
Hi Andrey,_ It's been about three days that we trying to solve the problem in integration, by integrating our code step by step, as you suggested!_ Hence, we noticed that the Plug-in disappears from the client, as son as the AI_Detector is declared in the code!_ However We couldn't step trough the debugging process to identify the problem. Hence, we're pretty sure that the problem does not from the library cause we already test it in other projects!Regards0
-
Hello,
I created a ticket. Please check your mail box for messages.
On resolution, the explanation will be put hereafter in a post.
0 -
Resume.
The instruction given on configuring Visual Studio to run server under debugger were correct.
Several corrections for initializing third party
Detectorclass.1. Declare a member inside DeviceAgent class in device_agent.h
2. Use correct syntax of the constructor of this class.
The resulting correct code is
Detector m_Detector = Detector(config_path, path);
Project compiled with no error.
0 -
Hi Andrey,
_ Thanks you so much for your help!
_ We succeeded to compile and generate the Plugin. Hence, the Plugin still does not appear in the panel. Do you have an idea on how to fix this issue?
_ Once again, thanks a lot!0 -
I'd really like to help you, but there is no enough data from your end.
> Do you have an idea on how to fix this issue?
Please, try to put questions specific, otherwise there is absolutely no possibility to determine what is the issue.
The algorithm of description is similar to this
1. I do these steps, putting this code in the file1, file2 etc.2. I expect to have <something to happen>
3. Actual result is <something has happened>
Here is the post with detailed instructions. Please, study it through.
https://support.networkoptix.com/hc/en-us/community/posts/360039290494-How-to-submit-a-support-request-0 -
Regarding technical aspects of your issue.
As I can judge after the remote session, there is no technical issue related to Metadata SDK itself.
I'm really sorry, but there is no magical pill making your plugin to have third part library built in.
You should work though SDK documentation, figure out how helper classes Plugin, Engine, DeviceAgent are organized.
After that carefully put your code INSIDE one of these classes.Now you are able to set breakpoints inside the plugin code, run the mediaserver under debugger in Visual Studio and step trough the plugin code in order to determine a place causing the issue.
Here are several more guidelines on plugin appearance.
There are two places where plugin information appears.
- "System Administration..."->"Plugins". Here information appears if
- the Plugin helper class instance was successfully created,
- Plugin::manifestString() returns a engineSettingsModel json object,
- the Engine helper class instance was successfully created.
- "Camera settings"->"Plugins". Here information appears if
- Engine::manifestString() returns a deviceAgentSettingsModel json object,
Here is a sequence of methods called by the Server when plugin is being loaded.
- The Server loads plugin dynamical library and invokes its entry point function createNxPlugin (see i_plugin.h), which returns a pointer to the nx::sdk::IPlugin interface class instance.
- The Server invokes IPlugin::createEngine using the pointer. createEngine calls the doCreateEngine virtualmethod of the interface class, which in turn - doObtainEngine. The IPlugin::createEngine returns a pointer to nx::sdk::analytics::IEngine interface class instance. The doObtainEngine is overridden in the Plugin helper class.
- The Server invokes IEngine::obtainDeviceAgent using the pointer. obtainDeviceAgent calls the doObtainDeviceAgent virtual method of the interface class. The IEngine::obtainDeviceAgent returns a pointer to the nx::sdk::analytics::IDeviceAgent interface class instance. The doObtainDeviceAgent is overridden in the Engine helper class.
- The Server invokes pushDataPacket method on retrieval of every packet. It is the general method for all types of plugins (analytics, video source, storage). It’ is the only method the Server knows. In case of Metadata SDK, pushDataPacket in turn invokes the doPushDataPacket virtual method of the interface class (see src/nx/sdk/analytics/i_consuming_device_agent.h). Network Optix has prepared the ConsumingDeviceAgent helper class (see src/nx/sdk/analytics/helpers/consuming_device_agent.cpp), which besides the reimplementation of doPushDataPacket offers several other methods in order to make plugin development more convenient and efficient: pushCompressedVideoFrame, pushUncompressedVideoFrame, pullMetadataPackets, pushMetadataPacket etc.
In general case, the helper classed Plugin, Engine, DeviceAgent are recommended to be used.
You could set breakpoints in the indicated methods as well as in the constructors of the Plugin, Engine, DeviceAgent helper classes to determine what is going wrong, what fails and restricts a plugin information to appear in the GUI.0 - "System Administration..."->"Plugins". Here information appears if
-
Here the algorithm that describes the issue:
1. I create this object, “Detector m_Detector = Detector(config_path, path)” in “DeviceAgent”. Then I compile the plugin without any problem. I copy the DLL in plugin folder "C:\Program Files\Network Optix\Nx Witness\MediaServer\plugins" and run VS in debug mode with several breakpoints;- I expect to that VS will stop at the breakpoints and the plugin will appear in the panel of my client;
- Actual result is :
a_ VS does not stop at the breakpoints (debug mode);
b_ and the plugin doesn’t appear in the panel
These two notifications I already reported them in my previous posts:
a_ https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011751293
b_ https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011558014
Regards,
0 -
Hello Andry
We sent you, by e-mail, a VS ".sln" with adequate configurations. We include the lib and the configuration files of the AI detector.
Regards
0 -
Guys,Could you please, kindly respond via ticket system? It is very difficult to follow several communication channel.I just lost your email.Please, duplicate your last response in the ticket.Thank you for understanding.0
-
_ Where can we find ticket system? Could you please provide us with a link to the "ticket system"?0
-
Hello,
Here is the result of our findings.
The
codeTestProjectyou've sent besides .lib files in thecodeTestProject\darknetfolder usespthreadCV2.dlllibrary, located incodeTestProject\x64\DebugandcodeTestProject\x64\Releasedirectories.
These directories are actually Output Directories, configured in your project. (See a screenshot)
Where
$(SolutionDir) iscodeTestProject
$(Platform) isx64
$(Configuration) isDebugorRelease
That means the resultingcodeTestProject.exeis placed in these directories during the build process and when is being executed, finds apthreadVC2.dllnearby and successfully loads it.
Please, make an experiment.
1. Open the codeTestProject project in the Visual Studio.
2. MovepthreadVC2.dllfromcodeTestProject\x64\DebugandcodeTestProject\x64\Release
3. Build the project.
4. Press F5.
You'll get an error.
This error is the root cause of why your plugin is not loaded and does not appear in the GUI. Plugin's dll just can't find and load thepthreadVC2.dll.
This failure is detectable by PluginManager message in the server log file2020-07-13 20:01:57.574 1fc0 INFO PluginManager(0x210a35403b0): Considering to load Server plugin [C:/Program Files/Network Optix/Nx MetaVMS/MediaServer/plugins/pthreadVC2.dll]
2020-07-13 20:01:57.602 1fc0 ERROR PluginManager(0x210a35403b0): Failed loading Server Plugin [C:/Program Files/Network Optix/Nx MetaVMS/MediaServer/plugins/pthreadVC2.dll]: [invalidLibrary]: No entry point function createNxPlugin() or old SDK createNXPluginInstance()
2020-07-13 20:01:57.603 1fc0 INFO PluginManager(0x210a35403b0): Considering to load Server plugin [C:/Program Files/Network Optix/Nx MetaVMS/MediaServer/plugins/sample_analytics_plugin.dll]
2020-07-13 20:01:57.638 1fc0 ERROR PluginManager(0x210a35403b0): Failed loading Server Plugin [C:/Program Files/Network Optix/Nx MetaVMS/MediaServer/plugins/sample_analytics_plugin.dll]: [cannotLoadLibrary]: Cannot load library C:\Program Files\Network Optix\Nx MetaVMS\MediaServer\plugins\sample_analytics_plugin.dll: The specified module could not be found.So, please, organize your plugin binaries as was advised above in this post https://support.networkoptix.com/hc/en-us/community/posts/360046685314/comments/360011516774, i.e. copy your plugin .dll and pthreadVC2.dll to one folder.
0
Please sign in to leave a comment.
Comments
26 comments