Is the Nx Cloud up? Visit our Status Page for the current health and performance of the Nx Cloud.

Status Page

Object Search using descriptions/AI

Joel Sabine

I would love to see the option to perform object search using natural language input where a user can type what they're looking for eg. ‘DHL truck’ or ‘Man wearing a red shirt and backpack’ rather than having to pick from the pre-defined attributes list.

The workflows I've seen operating this way are typically using an LLM to interpret the request and return initial results from the object database, then pass these objects to an AI image processing workflow to further extract information to determine complete matches. For example - using the ‘DHL Truck’ search input, the LLM would identify that trucks are an existing object type and can return all trucks from the object database and pass these into the AI workflow to determine if any are specifically DHL trucks before passing back to the user.

Whether this is something end to end developed by Network Optix, or just a matter of implementing the UI features required across the Desktop, Mobile and Web clients and allowing 3rd party plugin developers to fully implement in their own way…

4

Comments

4 comments

Date Votes
  • Miguel Câmara
    • Authorized Reseller
    • Great answers
    • Conversation starter
    • Idea generator

    Good suggestion

    I have seen this already being used in a camera manufacturer's VMS, and even just demonstrating this, before release, it was the cornerstone on convincing two customers.

    While this is still “new”, and not included in other VMS's as standard, it will bring out the awe reactions from the customers, which might trigger a decision making on the spot.

     

    1
  • Tagir Gadelshin
    • Network Optix team
    Edited

    hey
    We do want to work on this, but one caveat is cost. If we do it in the Cloud, that would either be a separate paid service or we'd need to require a dedicated GPU on-premises for vectorization. However, that would significantly increase hardware costs and limit our potential audience.
    Please let me know your thoughts on this. Other Cloud solutions that do this are quite pricey, so it might be that we have some options here. 

    1
  • Joel Sabine

    Hey Tagir Gadelshin ,

    I've been thinking about this a bunch and I can really see both sides as having valid use cases, so perhaps this is something that could be configured multiple ways within the platform itself depending on the available hardware? 

    For the cloud/SaaS approach, this would seem straightforward enough to manage as a service within Nx Connect, and this would allow that functionality to be offered at scale with minimal hardware on site. 

    I think there's definitely use cases though where handling this on prem is more ideal for customers - we've recently been deploying a lot more hardware with entry level GPUs like the NVIDIA RTX A400 so that we can run AI plugins like CVEIDA which offer far more capability than is generally available in camera (as well as the ability to run this at higher resolutions). These are the systems that would greatly benefit from these kinds of search capabilities, so while there is a greater hardware cost, the customers that benefit the most are already doing this. This would also apply for sites running the AI manager when that becomes more broadly available. Also, for smaller systems, Arrow Lake CPU architecture still offers a reasonably low pricepoint while including CPU/GPU/NPU - I think options in this space will become more broadly available as demand increases and the hardware catches up.

    Most of what I had seen so far across the industry was really only using cloud to support this feature, however more recently, we're starting to see NVR products include the required GPU to run locally - Eg. Hikvision AcuSeek, Unifi AI Key. This is also a standard/included features on some cloud platforms that aren't overly expensive - eg. Vivotek Vortex.

    I've seen this feature released across at least 10x different platforms so far (some of them still in beta though) and it definitely appears as though there's a lot of interest from users in how this can save time for investigations.

    2
  • Edward Dickey

    I definitely want this feature. I'd be using it on-premises only, not cloud. I have recent i7 and i9 video clients with RTX4060 that could use this. Hikvision, Dahua and Uniview have now all released their Large LLM search functions. 

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post