Scailable Nx AI Manager Custom Trained Object Detection Model (YOLOv8 or YOLOv10) Doesn't Predict
In Progress
Summary
We tried to integrate our YOLOv8 and YOLOv10 object detection model that has been fine tuned with our custom data such that it can predict custom objects. We exported it to onnx using the sclblonnx python package. Upload and assign the model to our server and start the runtime but the plugins or the model doesn't predicts, no objects detected, no bounding box written, no event screenshoot.
Environment
-
Nx Witness Version: Desktop Client version 6.0.0.38488 R4 (7cc6b3c72284).
Built for windows-x64 with msvc.
OpenGL version: 4.6.0 NVIDIA 551.86.
OpenGL renderer: NVIDIA GeForce RTX 3070/PCIe/SSE2.
OpenGL vendor: NVIDIA Corporation.
OpenGL max texture size: 32768. - Client OS: Microsoft Windows 10 Pro - 10.0.19045 Build 19045
- Server OS: Ubuntu 22.04.3 LTS
- Network Topology: 1 servers, 1 cameras (rtsp streamed using mediamtx), 1 or more clients
- Special features: None
Reproduction Scenario
- Train YOLOv8 or YOLOv10 object detection model with our custom data such that it can predict custom objects. We tested the fine tuned model with our own inference pipeline, it works and gives prediction.
- Export the model to onnx using sclblonnx python package. Following this instruction https://nx.docs.scailable.net/for-data-scientists/importing-models/common-models/yolov5-and-yolov8
Yes, we did
Call bash export-to-onnx.sh with the appropriate arguments (in particular the first and last arguments: model name, and input size).
Update the complete_onnx.py script to add the classes your model is trained on. - Upload the model to the Nx AI plugin admin server management https://admin.nx.scailable.dev/system/<id>
Assigned it to our server. - Open our device settings, enable the plugin and let the Nx AI plugin installing
- Once it's installed, select our model, start the runtime, apply
Actual Behavior
The view camera should've write the predicted bounding box, and send the screenshoots to the right detected object area. But it doesn't
Expected Behavior
The view camera write the predicted bounding box, and send the screenshoots to the right detected object area.
Yes, we did make it happens, but the one that works is not our custom fine tuned model. The one that works is a YOLOv8n base pretrainde model on COCO dataset. Fully following this instruction https://nx.docs.scailable.net/for-data-scientists/importing-models/common-models/yolov5-and-yolov8
Additional Information
Since we can add multiple object detection model on the plugin. We tested the multiple model (the one that works which is YOLOv8n base and my custom models) uploaded and enabled as well and the behavior is now none of the detections is shown or working.
There is a default catalogue of model brought as we install the Nx AI Plugin, which by default as I enable it, it doesn't work as well on our client.
The YOLOv8 and YOLOv10 object detection model is by Ultralytics
-
Hi, thank you for question.
Since you got a model up and running, I'm assuming that the device and the installation works well, and the problem lies with the specific models.
Did you have a look at the ONNX Requirements? https://nx.docs.scailable.net/for-data-scientists/onnx-requirements
Specifically note the ONNX version, static input sizes and named outputs.If your model adheres to these requirements, but still doesn't work, is there a way you can make this model available to us so we can have a look at what the problem might be?
-
Hi, thank you for your reply.
Recently we managed to make our custom fine tuned YOLOv8 object detection model works well, but for the YOLOv10 we managed to make it work as well but not really well. We understand that the model is very new, recently published, we think that the main reason is because when we do so.check(g) for the YOLOv10, it has quite a lot operator that is not supported yet.
Is it normal that the model can run on nx despite many operators are not supported yet?
Yes, The main problem is somehow solved, but if you guys have insight or tested on the YOLOv10 performance, please let us know. Thank you.
Above is the YOLOv8 model. Our custom fine tuned model.
Above is the YOLOv10 model. This is the base model by them YOLOv10n.pt that we convert to onnx
The YOLOv10 model that we tested is available on this link -
Thank you for the reply and confirmation.
Regarding the support for the operators, when I checked the YOLOv8 model, it does have some unsupported operators as well.
Can the team check as well for the YOLOv8 model? As for now what we are aware is that even though there's unsupported operators, the model still can work but it may affect its performance. Hopefully it is fixed as well for YOLOv8 so that it performs at its best as I believe the model has already been on the attention as there's an available guide for it (link)
Below is the snippet to check the onnx model.import sclblonnx as so
g = so.graph_from_file("best-complete.onnx")print(so.check(g)) -
Hi Benedictus Visto Kartika,
Thanks for pointing out to the operators validation error.
We'll update the "sclblonnx" package in the coming days/weeks to suppress those errors, as the latest version of the Nx AI Manager supports any Vision ONNX model with IR version up to 17.
When we support the Yolov10 model, we'll provide a guide that shows how to correctly generate an ONNX that works with the AI Manager.
Keep you posted. -
Hi Benedictus Visto Kartika, you can find a tutorial for deploying Yolov10 using the Nx toolkit here: https://github.com/scailable/nxai-model-to-onnx
Please sign in to leave a comment.
Comments
6 comments