Can We Build NX AI Manager Custom Postprocessor That Has AI Model and Accelerated By Nvidia CUDA?

Answered

Benedictus Visto Kartika

October 01, 2024 08:44

Greetings,
Thank you for the great open NX Meta AI Manager, we are so thrilled to integrate our AI using the NX AI Manager.

We managed to build a face recognition use case for CCTV camera using NX AI Manager. We have our base YOLO object detection model that detect face, and we build custom face recognition postprocessor following this example https://github.com/scailable/sclbl-integration-sdk/tree/main/postprocessor-python-image-example and built and install it using cmake

The question is that it seems that I haven't manage able to run the postprocessor on GPU, I'd like to confirm is it possible to build custom postprocessor with AI that can run with GPU acceleration? Is there any specific model format that needs to be followed (e.g. ONNX, PyTorch, TensorFlow)? My model is currently in ONNX format (Yes I have tested outside of custom postprocessor that it can run using GPU)

Environment:
Python 3.8 (in building the postprocessor)
Nx Meta 6.0.0.39503
Nx Client 6.0.0.39503

1

• 

  Josef Joubert

  October 03, 2024 13:14

  Hi, thank you for your question.

  Very cool that you built some facial recognition software and got it to work.

  The AI Manager places no restrictions on what a custom postprocessor can do. The postprocessor runs as a completely separate application which can use any API's/tools/hardware it needs to. If running the postprocessor outside the AI Manager runs on CUDA, then it should be able to do the same inside the AI Manager.

  I suspect the reason your custom post-processor doesn't run on GPU in the AI Manager could be due to how it's compiled. For convenience all the examples are compiled into binaries. The Python examples are compiled using PyInstaller. We don't take special care to ensure CUDA tools work after compilation.

  You could try just copying the Python script(s) and adjusting the "Command" field in the "external_postprocessors.json" to start the script with Python, for example "python3 <your_script>.py".

  Another option is to write a .spec file, which is read by PyInstaller, which can compile your Python script correctly to use CUDA libraries.

   

  Let me know how it goes and what you manage to achieve!

  0
• 

  Benedictus Visto Kartika

  October 04, 2024 02:19

  Well noted and thank you for confirming that custom postprocessor can work on CUDA. Will ensure and check my compiler script and make sure my CUDA version is compatible and in the same environment as well. Thank you once again

  0

Please sign in to leave a comment.