Can't seem to get the track information in executeAction
Answered
Hey, I'm using the NX Meta SDK and i've implemented a plugin following along from the stub analytics plugin provided as a sample.
Based on the documentation, there are two ways to push metadata to the NX server, via 'pushMetadataPacket' or implementing your own 'pulMetadataPackets' function. The stub analytics plugin implements the pullMetadataPackets function, however I've chosen to use the pushMetadataPacket since I don't need to interact with the frame at all for the purposes of my plugin.
I'm able to push metadata successfully to the NX server and it appears on the overlay in the NX MetaVMS. The issue I'm having however, is I'm trying to implement an 'action' that can be triggered on a frame with its associated metadata. The documentation says - if your solution needs context actions, declare them in Plugin's manifest and implement function nx::sdk::analytics::Engine::executeAction().
I have implemented this function, and my particular action needs access to the attributes, which I believe are located by accessing the objectTrackInfo parameter - I've based this on what I get by printing the outputs of a working action in the stub analytics plugin in the SDK. My objectTrackInfo seems to not have anything on it though, am I wrong in assuming this is how I would go about getting the object metadata that is attached to that trackId?
Here is the code related to this that hopefully is helpful to diagnose my issue:
In my device_agent.cpp file I have this:
void DeviceAgent::pushObjectMetadata(int64_t timestamp, std::string typeId, nx::sdk::Uuid trackId, nx::sdk::analytics::Rect boundingBox, std::vector<nx::sdk::Ptr<nx::sdk::Attribute>> attributes, int duration)
{
nx::sdk::analytics::ObjectMetadata *objectMetadata =newnx::sdk::analytics::ObjectMetadata();
objectMetadata->setTypeId(typeId);
objectMetadata->setTrackId(trackId);
objectMetadata->setBoundingBox(boundingBox);
objectMetadata->addAttributes(attributes);
nx::sdk::analytics::ObjectMetadataPacket *objectMetadataPacket =newnx::sdk::analytics::ObjectMetadataPacket();
objectMetadataPacket->setTimestampUs(timestamp);
objectMetadataPacket->setDurationUs(duration);
objectMetadataPacket->addItem(objectMetadata);
nx::sdk::analytics::VideoFrameProcessingDeviceAgent::pushMetadataPacket(objectMetadataPacket);
}
In my engine.cpp file I have this:
nx::sdk::Result<nx::sdk::analytics::IAction::Result> Engine::executeAction(
conststd::string &actionId,
nx::sdk::Uuid trackId,
nx::sdk::Uuid deviceId,
int64_ttimestampUs,
nx::sdk::Ptr<nx::sdk::analytics::IObjectTrackInfo> objectTrackInfo,
conststd::map<std::string, std::string>¶ms)
{
std::string actionUrl;
NX_PRINT << __func__
<<"\ntrackId: "<< trackId
<<"\ndeviceId: "<< deviceId
<<"\nobjectTrackInfo: "<<objectTrackInfo->track()
<<"\nbestShotObjectMetadata: "<<objectTrackInfo->bestShotObjectMetadata();
if (actionId == kViewProfileAction)
{
actionUrl = "some url I will construct based on the track info";
}
else
{
returnnx::sdk::error(nx::sdk::ErrorCode::invalidParams, "Unsupported actionId");
}
returnnx::sdk::analytics::IAction::Result{nx::sdk::makePtr<nx::sdk::String>(actionUrl)};
}
The executeAction function isn't currently doing anything, but when i print out the outputs of the track and bestShotObjectMetadata i get 0 in contrast to the stub analytics plugin which has a track.
Any ideas where I'm going wrong?
Thanks, Sebastian
-
Official comment
Hello Sebastian,
It is important to understand the structure of the mement when executeAction is executed.
When user right-clicks an object displayed in Nx Client, he is given possibility to select an action which was declared in manifest string of plugin engine, along with additional data, of which necessity to be sent can be declared in manifest, as well.
That means, if you want the plugin to receive extra data, this data should be declared in plugin engine's manifest.
In your case, if you want to get 'objectTrackInfo->bestShotObjectMetadata' you need to declare 'needBestShotObjectMetadata' in 'requirements' section of manifest. For example,
"requirements": {
"capabilities": "needBestShotObjectMetadata"
}For greater details see 'objectActions' in samples/stub_analytics_plugin/src/nx/vms_server_plugins/analytics/stub/engine.cpp
Current version of your manifest does not contain such declaration, that's why executeAction does not receive 'objectTrackInfo'. This fact is indicated by 'mediaserver[20606]: objectTrackInfo: 0' in log file.
Since objectTrackInfo is a pointer to interface the good practice is to check the pointer before referencing it.
See implementation of 'executeAction' in the same engine.cpp. Here is a quotation:
if (actionId == "nx.stub.addToList")
{
messageToUser = std::string("Track id: ") + UuidHelper::toStdString(trackId) + "\n\n";if (!objectTrackInfo)
{
messageToUser += "No object track info provided.\n\n";
}
else
{...
-
Hi Sebastian,
That's a good question, we will talk to developers and figure out what to do.
0 -
Hello Sebastian,
Could you please do the following?
1. Make sure you have video recording turned on.
2. Turn on debug logging on the Server while your are debugging your plugin.
Send us the logs, please.
3. "The documentation says - if your solution needs context actions, declare them in Plugin's manifest and implement function nx::sdk::analytics::Engine::executeAction()."
Send us manifestString() from device_agent.cpp and from engine.cpp, please.
0 -
Sorry for the late reply
1. Video recording is turned on
2. Debug logging is turned on
3.std::string Engine::manifestString() const
{
// clang-format off
returnR"json({
"objectActions": [{
"id": ")json" + kViewProfileAction + R"json(",
"name": "View profile",
"supportedObjectTypeIds": [
")json" + kFaceObject + R"json("
]
}]
})json";
// clang-format on
}0 -
Is anyone able to help me on this still? Maybe this extra information could help.
nx::sdk::Result<nx::sdk::analytics::IAction::Result> Engine::executeAction(
This gives me back this log
conststd::string &actionId,
nx::sdk::Uuid trackId,
nx::sdk::Uuid deviceId,
int64_ttimestampUs,
nx::sdk::Ptr<nx::sdk::analytics::IObjectTrackInfo> objectTrackInfo,
conststd::map<std::string, std::string>¶ms)
{
std::string actionUrl;
NX_PRINT << __func__
<<"\ntrackId: "<< trackId
<<"\nobjectTrackInfoRaw: "<< objectTrackInfo
<<"\nparamsRaw: "<<¶ms
<<"\nobjectTrackInfo: "<<objectTrackInfo->track()
<<"\nbestShotObjectMetadata: "<<objectTrackInfo->bestShotObjectMetadata()
<<"\nformattedTrackInfo: "<<std::string("Object track info:\n") +" Track: "+objectTrackToString(objectTrackInfo->track(), trackId);
returnnx::sdk::analytics::IAction::Result{nx::sdk::makePtr<nx::sdk::String>(actionUrl)};
}
mediaserver[20606]: [unnamed_lib_context_engine_{f7c76ed9-f325-5498-56cf-303011d9cb16}] executeAction
mediaserver[20606]: trackId: {AB7BB8ED-7389-4965-89CF-FE2CFC9731DF}
mediaserver[20606]: objectTrackInfoRaw: 1
mediaserver[20606]: paramsRaw: 0x7f1a4c0c6ff0
mediaserver[20606]: objectTrackInfo: 0
mediaserver[20606]: bestShotObjectMetadata: 0
mediaserver[20606]: formattedTrackInfo: Object track info:
mediaserver[20606]: Track: null
Notice how i have nothing on this object track Info. This is different to the stub analytics plugin provided, where the attributes can be accessed. How do I get that information, I'm using pushMetadataPacket instead of pullMetadataPacket which is the only culprit I can think of. I'm not experienced enough in C++ to be able to understand the inner workings here, so any help would be greatly appreciated.
Something else I saw with the debug logs that is interesting when I click an action, is I get this:
2019-12-10 14:18:52.155 19601 DEBUG QnExecuteAnalyticsActionRestHandler(0x7f59801247b0): The action type facerec.viewProfileAction has no addittional requirements.Track, best shot frame and position are not going to be fetched.
It's as though I'm missing some key piece of code that will push the track info to the executeAction function.
Thanks again, Sebastian0 -
Thank you very much, this resolved my issue! Appreciate the detailed response
0
Please sign in to leave a comment.
Comments
6 comments