issue of Handling bestShotImageData via objectTrackInfo within objectActions (null)
Answered
According to the nx sdk documentation, I expected data to be returned from bestShotImageData when an image is designated for the best shot.
"Pointer to the track best shot image data provided by the Plugin. If the best shot was provided in the form of a URL, the pointer returned by this method points to the data of the image that the server obtained by this URL. If the Plugin provides neither binary image data nor its URL, null is returned."
my designation of the best shot was successful, and the designated image is correctly displayed in the right-side object panel.
However, while running the Object actions, I logged the results returned by the bestShotImageDataFormat() and bestShotImageDataSize() functions, and as you can see, they either return no values or 0.
Yet, when I check the track ID through the API, I find that a best shot in the form of base64 does exist. (By the way, have you informed me that the data type was base64? It’s not bad, but I was surprised.)
Am I missing something? Please let me know, thank you.
-
Hello Vikram,
We kindly ask to provide more details for the issue you're dealing with.
my designation of the best shot was successful
Please specify how the best shot was provided to the server. Best case scenario if you provide a minimal patch to the Sample or Stub plugin code that would replicate the behavior.
I'm asking that because based on the output you attached there is a chance that no actual image was passed to the Media Server as a best shot which is why you get null for type and 0-size.
BUT! You did pass the timestamp attribute value which made server automatically decode the video chunk and extract the next closest frame to the requested timestamp value. Which is why the result timestamp is slightly different from the requested. And in fact this is the expected behavior.
-
device_agent.cpp
cv::Mat rgbImage = cv::Mat(
image_height,
image_width,
CV_8UC3,
(void*)image_data,
(size_t)0);
std::vector<uchar> buffer;
cv::imencode(".jpg", rgbImage, buffer);
std::vector<char> charBuffer(buffer.begin(), buffer.end());
bestShotPacket->setImageDataFormat("image/jpeg");
bestShotPacket->setImageData(charBuffer);
engine.cppconst char* imageData = objectTrackInfo->bestShotImageData();
int imageSize = objectTrackInfo->bestShotImageDataSize();
if (imageData && imageSize > 0) {
cv::Mat data(1, imageSize, CV_8UC1, (void*)imageData);
image = cv::imdecode(data, cv::IMREAD_COLOR);
cv::imwrite("cv_enrollshot.jpg", image);
}I sent the necessary amount of image data from device_agent.cpp.
And I intended to receive this in engine.cpp.
The reason I deemed this successful is that instead of the object being depicted as a videoframe with a background and highlighted with a bbox, only the specified image I designated was displayed. (Do you understand what I mean?)
-
Any news? Sergey Yuldashev
-
Hello Vikram,
I'm afraid I don't understand what you mean. Could you please shed some lint on what is your objective? What is the purpose you need the besthost image data in the Engine class? Could you share a screenshot of the desired result?
JFYI, the best shot image (if set) is displayed in the right-hand panel when the detected object is being displayed. There is no way to tell the Server what image to display on the scene instead of the frame extracted from the video archive by the timestamp.
-
I expected that I would be able to retrieve the image sent in the bestshot packet using this function (
bestShotImageData) after reading its description. Are you saying that this is not possible? -
All these methods are supposed to be invoked by the Server.
Vikram, I'm trying to help you and to resolve your trouble in aheiving your goal, but you're not answering the questions I'm asking.
Instead, you're posing additional questions. Answering them is really difficult without knowing your objective.
Could you please shed some lint on what is your objective? What is your business scenario? What is the purpose you need the best shot image data in the Engine class? Could you share a screenshot of the desired result?
Having that information from you and having developer team by my hand, I could find and recommend the most suitable solution for your case.
-
I appreciate your efforts so far. I think misunderstandings may have grown because English is not my native language. I am working based on the metadata SDK stub example, specifically with the bestshot example and object action example. This is part of the code provided in your examples:
[object action example: engine.cpp]
Result<IAction::Result> Engine::executeActionWithRequirements(Uuid trackId,Uuid deviceId,int64_t timestampUs,Ptr<IObjectTrackInfo> objectTrackInfo){IAction::Result result;
std::string message = "Message generated by the Plugin:\n"" Track id: " + UuidHelper::toStdString(trackId) + ",\n"" Device id: " + UuidHelper::toStdString(deviceId) + ",\n"" Timestamp: " + nx::kit::utils::format("%lld us", timestampUs) + ",\n"" Object Track Info:";
if (objectTrackInfo){message += "\n Best shot frame: "+ uncompressedVideoFrameToString(objectTrackInfo->bestShotVideoFrame()) + "\n"+ " Best shot metadata: "+ timestampedObjectMetadataToString(objectTrackInfo->bestShotObjectMetadata());}else{message += "null";}
result.messageToUser = makePtr<String>(message);
NX_PRINT << "Executing an Action with requirements, returning a message: "<< nx::kit::utils::toString(message);return result;}[bestshot example: device_agent.cpp]
Ptr<IObjectTrackBestShotPacket> DeviceAgent::generateImageBestShot(Uuid trackId){auto bestShotPacket = makePtr<ObjectTrackBestShotPacket>(trackId,m_lastFrameTimestampUs);
bestShotPacket->setImageDataFormat(m_bestShotGenerationContext.imageDataFormat);bestShotPacket->setImageData(m_bestShotGenerationContext.imageData);
return bestShotPacket;}Looking at these two, is it really so confusing to think that I could retrieve the data from generateImageBestShot in the engine?
That was my thinking, but ultimately, the desired result is as follows:
I want to pass the bestshot image (data) I generated directly to an external API through object action. The videoframe is not sufficient because it doesn't contain the bounding box area I need, so it may not work as intended. Perhaps it would be enough if I could just get the base64 image information from the track info I initially uploaded.
I hope you understand well now. Thank you.
-
Hello Vikram,
Thank you for details and explanation.
Several notes, regarding the code snippets you shared and the plugins you referred to.
The Stub plugin and all sub-plugin don't perform any image processing, they simulate image processing. The purpose is to demonstrate how to use SDK's classes, interfaces, method to get advantage of SDK's features.
generateImageBestShot() generates best shot image data to be passed to the Server. The Server saves it to and stores in the database. The best shot data cam be retrieved afterwards via Server API or in the Metadata SDK, as it's demonstrated in the object action sub-plugin.
The method
Result<IAction::Result> Engine::executeActionWithRequirements(
Uuid trackId,
Uuid deviceId,
int64_t timestampUs,
Ptr<IObjectTrackInfo> objectTrackInfo)is invoked by the interface method Engine::executeAction()
Result<IAction::Result> Engine::executeAction(
const std::string& actionId,
Uuid trackId,
Uuid deviceId,
int64_t timestampUs,
Ptr<IObjectTrackInfo> objectTrackInfo,
const std::map<std::string, std::string>& params)The method definition states, the SERVER PASSES a pointer to the IObjectTrackInfo class instance, initiated with object track info data. For that, the Server has to have this data in the database. That, in turn, means, that a plugin has to pass this object track info to the Server before the executeAction() is called.
In other words, if there is no track info (including best shot image data) in the database, the Server will take the first object occurrence of the track as the best shot image.
One more thing. To make the Server pass Ptr<IObjectTrackInfo> objectTrackInfo the plugin has to specify requirements in the Engine's manifest. See the "capabilities" flags.
{
"id": ")json" + kObjectActionWithRequirementsId + R"json(",
"name": "Stub: Object Action with requirements",
"supportedObjectTypeIds": [ ")json" + kObjectTypeId + R"json(" ],
"requirements":
{
"capabilities": "needBestShotVideoFrame|needBestShotObjectMetadata|needFullTrack",
"bestShotVideoFramePixelFormat": "yuv420"
}
}
Please sign in to leave a comment.
Comments
8 comments