Incorporating image processing delay
In Progress
I'd like to make our plugin cooperate with backend server, which processes image frame.
This backend server causes processing delay time. I'm going to deal with this delay correctly.
But now, processing delay seems not reflected correctly. I want to ask about this point.
I've already done implementation to test.
I'm using pushUncompressedFrame, and added codes to cooperate with backend within this method.
However while testing, while server processing image, new frames are incoming.
I'm afraid that this causes inconsistent result.
What should I do to deal with backend processing delay consistently?
-
Thank you for your questions. Unfortunately, the description you provided contains several undefined terms, could you please provide more clarity and detail?
1.Backend Server:
•When you mention the “backend server,” what exactly are you referring to?
•Is it your application, the Nx Media Server, or something else? The term is unclear in this context.
2.Delay and Reflection:
•You mentioned dealing with a delay. What steps have you already taken to address this?
•Additionally, when you say “not reflected correctly,” what exactly do you mean?
•Could you elaborate on what you are observing versus what you expect to happen?
3.Frame Processing and Inconsistency:
•Your comment about “while the server is processing an image, new frames are incoming” describes normal behavior for most applications. What does this mean exactly and why this cause an issue to you? What specific inconsistencies are you encountering?
At this point, it’s challenging to provide assistance without a clear understanding of what you’ve implemented, observed, or the exact use cases and goals you are trying to achieve.
We kindly suggest summarizing the exact use cases, the procedures or implementations you have already performed, and your desired outcomes. Providing more detail will help us better understand the situation and provide more targeted assistance.
Thank you for your understanding, and we look forward to your clarification.
-
Thank you for reply.
Certainly my post has contained unclear matters.
1. Backend Server
This server is python HTTP API server, which processes AI inference. It communicates with Nx plugin via HTTP protocol.
Nx Plugin posts (image, timestamp, etc) to server, and server response inference result.
2.Delay and Reflection:
For testing, I added dummy sleep for python API implementation. This is because processing image takes some time.
3.Frame Processing and Inconsistency:
We visualize the result got from Object Detection AI, so want to make sure that "visualized result is compatible with shown image". It is okay if that compatibility is filled in record.
-
Hi Fujiwara Daichi,
Unfortunately, the explanation seems not very clear to us.
Can you please explain what you exact want to do and what you have done?
Also, there are a few more terms coming up --This server is python HTTP API server, which processes AI inference. It communicates with Nx plugin via HTTP protocol.
So you issue in on your HTTP Server? Or what and where is the issue?
Sorry we are confused to the post at the moment, can you provide the whole story and implementation that you were doing and explanation?Nx Plugin posts (image, timestamp, etc) to server, and server response inference result.
What (and which) is the plugin referring to?
We visualize the result got from Object Detection AI, so want to make sure that "visualized result is compatible with shown image". It is okay if that compatibility is filled in record.
Sorry, we still don't quite understand what your issue is and where the issue happened exactly?
Can you provide the screen recording or elaborate what you have seen and what was the exact issue please?
-
Nx Plugin is our own plugin, not an official plugin.
Background:
We want to integrate our AI with Nx Witness. Like, AI inference result is shown on Nx Witness GUI.
Where, AI inference result is customized in according with our AI output format.
This result contains boundingbox coordinates, confidence, etc...
We setup this AI as HTTP server, by using some specific API, we get result.
We startup AI server at first, and then Nx plugin, then we are going to integrate them.
Done and doing:
I developped dummy server, accept specific API. Out plugin posts data using this API, like
http://localhost:5000/post_image?image=...
Using got result(detected object and additional information), we built metadata packet.
Then show this result to GUI.
Now I'm testing with processing delay and http communication with dummy server.
--------------------------------------------------------------------------------------------------------------------------
The problem is only one point. Server side description are not important.
I want to make explicit correspondence between "metadata packet" and "video frame".
There is not any problem on server side. Problem is
- It takes time to deal with image.
- while processing image, new image is comming
- visualized result is shown as a result of new image after
Flow:
- My goal is this: (timeline goes from left to right)
frame1(plugin) -> inference(server) -> metadata1(plugin) -> GUI(Nx Witness)
- But currently it seems like:
frame1(plugin) -> inference(server) ..(processing).. metadata1(plugin)↓
frame2(plugin) -> GUI(Nx Witness)
frame3(plugin) -> metadata1(plugin) -> GUI(Nx Witness)
-
Hi Fujiwara Daichi,
Thanks for the update. If we understand correctly, your question is the metadata(ex: Boundary boxes) is not attached to the proper object maybe delay or offset can be seen?
The video frames will continuously be sent to the plugin as you have requested. Even if processing takes time, you can still assign the correct timestamp for each frame, ensuring that metadata(after process by external application or reosurce) is accurately attached to the specific frame with the exact timestamp.
Could you verify if you have set the correct trackId and timestamp? You can refer to the MetaSDK sample code for guidance, specifically the implementation in STUB/Object Detection/deviceAgent. This example provides a useful reference for your implementation.
-
Thank you for defail reply.
your question is the metadata(ex: Boundary boxes) is not attached to the proper object maybe delay or offset can be seen?
I thought that it "could" be so. I think I'll make some debug to understand situation correctly.
The video frames will continuously be sent to the plugin as you have requested. Even if processing takes time, you can still assign the correct timestamp for each frame
Fine, it is okay if correspondence between timestamp and result is ensured.
Please sign in to leave a comment.
Comments
6 comments