Is the Nx Cloud up? Visit our Status Page for the current health and performance of the Nx Cloud.

Status Page

Integrating custom H/W with AI accelerator on NX Meta for inference.

Planned
Base

Hi, I'm working on invoking AI Accelerators on custom hardware for inference. I came across the OAAX Framework on the NX website and understand that it allows us to integrate a custom runtime and board-specific model with the toolchain. However, I couldn't find information on how to integrate the OAAX Runtime/Toolchain with NX Meta.

Am I missing something? Could you please share any references or guidance for integrating the OAAX Runtime/Toolchain with NX Meta for custom hardware?

Also, can we include Pre/Post Processing for the inference in the OAAX Framework ? If not, where can we incorporate the same. Please help!

0

Comments

3 comments

Date Votes
  • Base

    Thanks for the response.

    I clearly understand use of OAAX as wrapper to create a standard mechanism to build models and execute it on specific Hardwares.
    I also understand that I need to write a sample application to make use the OAAX compliant runtime.
    I still have a few questions:
    1. In the link https://nx.docs.scailable.net/ai-accelerators-support/introduction-to-oaxs, It is stated :The Nx AI Manager accelerates AI models inference using various built-in runtimes, each of which is dedicated to a specific platform and AI accelerator (CPU, GPU, NPU, etc.). These runtimes are seamlessly integrated within the AI Manager, facilitating effortless utilization for individuals employing new runtimes and harnessing various forms of AI acceleration.
    Any runtime/toolchain combination adhering to the Open AI Accelerator eXchange (OAAX) standard is compatible with the Nx AI Manager, enabling straightforward substitution of any existing installed runtime with new one.  

    Based on this, I assume that once a runtime/toolchain is made compatible with OAAX, it can be integrated into the NX AI Manager to run inference utilising the AI Accelerator. Could you confirm if this understanding is correct? If so, could you please share the details on the necessary modifications to be done in the NX AI Manager to integrate an OAAX compatible runtime/toolchain?

    2. Regarding the 'NX Meta (Optional)' step: You can use NX Meta to track the model you used in the OAAX process, but it doesn't directly control the execution on your custom hardware. You would likely register the original model in NX Meta, not the compiled output.

    From the above points, What exactly does 'track' in this context mean ? Could you please elaborate on its purpose ?

    0
  • Base
    Edited

    Hi, can you please provide any guidance on the above questions ?

    0
  • Ayoub Assis
    • Network Optix team

    Hi, 

    It's great to hear that you're working on integrating your custom AI accelerator with the Nx software. Let me first provide a brief summary of OAAX and how Network Optix makes use of it.

    As you already learned from the documentation, OAAX is a standard/specification that details how an AI accelerator can be used by a host program running on CPU to offload the heavy computation of running AI models from CPU to GPU/NPU/XPU. 
    An ONNX model, before running on an AI hardware, needs to be optimized and compiled, by a conversion toolchain implemented specifically for that AI hardware, into a format compatible with the hardware.
    The compiled model runs on the device with the AI hardware using a runtime to OAAX's runtime interface.

    That said, for the Nx AI Manager to support an AI hardware XPU, it needs an OAAX runtime and toolchain pair implemented specifically for XPU. The two artifacts are then integrated as part of the Nx AI Manager package and the Nx AI Cloud.

    Hope this provides a clear understanding of how OAAX is adopted by Network Optix.

    Now, here are my answers to your questions.

    1. I couldn't find information on how to integrate the OAAX Runtime/Toolchain with NX Meta:
      The Nx team is the one responsible for integrating the runtime and toolchain with Nx Meta. There are proper channels and procedures for doing that. I (or someone from Nx) will get back to you later about this.
    2. can we include Pre/Post Processing for the inference in the OAAX Framework ? If not, where can we incorporate the same.
      It's possible to incorporate custom Pre/Post Processing in the AI Manager depending on your model. You can find more information in this documentation section.
    3. I assume that once a runtime/toolchain is made compatible with OAAX, it can be integrated into the NX AI Manager to run inference utilising the AI Accelerator.
      Please refer to the answer given for question 1.
    4. What exactly does 'track' in this context mean ? Could you please elaborate on its purpose ?
      Can you clarify your question.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post