Problems with Smart Motion Search
Answered
One out of four cameras stopped generating motion events. Motion is server-side for all cameras, two cameras do not have working second stream, so motion is done on the primary stream for them.
The malfunctioning camera was one of those two, for what its worth.
The afflicted camera would only show motion up until a certain date, about a week ago, and showed no motion subsequent, even when motion of obviously present. Selecting "Reindex Archive" did not help.
Stopping and restarting the server fixed things - motion is again seen, both in realtime [red mesh indicator] and for searching.
- How do I generate motion information for searching for the week where motion data was broken?
- Why/how did this happen? It's a bit alarming - no logs indicate anything unusual, other than a number of "the usual" dropped RTP packet notice.
As an aside, Smart Motion Search is amazing and I love it. I do wish for two enhancements (three if there's currently no way to fix missing motion information):
- I'd like to set separate motion masks for event generation, and can't figure out if this is possible. ie one small low sensitivity area for a door [to trigger Pushover, say], another small medium sensitivity area for an email trigger, and the whole camera at medium sensitivity for after-the-fact motion searches.
- I'd like to be able to set "motion spatial resolution", essentially to scale the stream down by a given factor before motion processing. This would address eg Primary Stream Only cameras, greatly improving CPU usage - perform motion on a scaled down stream, where the scaling is down server side rather than on the camera
-
Official comment
Rick,
We are aware of the issue with the motion detection on some camera models (especially Uniview). This is caused by camera sending some broken packets which are causing time desynchronization between the motion metadata and the actual video. The more broken packets are sent, the more the delay is and at some point the motion detection stops working completely.
If that's the case, the issue is known and we will provide you the patch. However, unfortunately the motion data is lost and cannot be recovered.
If it's not the case, then we may need a remote access information. If so, we will convert the topic into a support ticket and continue communication there.
Can you please let us know which camera model is causing the issue AND the software build number?
-
As for your feedback regarding smart search improvements:
1) We do not support separate motion masks so far. Please post this to "New Feature Ideas" for consideration.
2) I'm not sure is this makes much sense to me (or I didn't get you correctly). To do motion detection server actually decodes the stream from a camera (High-Res in your case). "Motion spatial resolution" that you suggest will force server to decode the High-Res stream AND transcode it to a Low-Res stream which will cause even more CPU load.
All in all, thank you for your suggestions!
0 -
The camera is a Vicon V962D-N312, running the latest VCN01.2.1986.96 firmware.
NX Witness is 4.0.0.29987
The Vicons I have are both using wireless interfaces [the cameras themselves are not wireless] which sometimes suffer from connection issues, so that may also be related.
The time desynchronization theory makes a lot of sense, and would explain why it started working again by just restarting the server. Let me know how to proceed on the patch.
I'm not suggesting a re-transcode, which I agree would be needlessly slow. What I'm suggesting instead is that there be a spatial downscale factor available, which affects motion processing. With a "downscale 4", we would scale a 1920x1080 image down to 480x270 before motion processing. This would result in 16x fewer pixels on which to perform motion processing, and should offset the cost of the downscale. No transcode, the downscale would be done as part of the processing chain.
Hopefully that makes more sense.
I will post the motion mask to "New Feature Ideas" as requested.
0 -
Rick,
Let's start with the fix
Please try to update to the latest patch:
In-Client Upgrade:
Build Number: 30617 Password: l2jy93 Please observe for several days if the issue is fixed.
If the issue persists, we will ask you to port-forward the camera for investigation.
Regarding spatial downscale we will get back to you shortly.
Thank you
0 -
OK, updated and will monitor
Thank you!
Cheers
0 -
Everything was fine until this morning, when there was a power failure.
Server has UPS, problematic camera does not.
As of an hour after the power fail, that camera is once again not sensing any motion. I can walk around in front of it and i no longer see the motion areas highlit.
Changing nothing and simply restarting the server fixed things.
0 -
Hi Rick,
If there is an option of providing our team access to Teamviewer to your system, I could convert this topic into a support ticket so our support team can investigate the issue.
Please let us know what the options are.
0 -
I'm afraid that is not possible, no remote access can be provided
One thing I have done with other VMS vendors in the past has been to send either a rtmp recording from VLC of the camera or a wireshark capture of VMS/camera packets. Let me know if either of those would be helpful
From a non-camera-specific point of view, I do feel like there's a meta-issue here, which is that the system has decided to no longer compute smart-motion data, but is providing neither user indication, loggable alerts, or the ability to restart/resync smart motion. One solution would be an alert, and the possibility of setting up a rule to recover when such an alert is thrown.
0 -
...also, although I hinted at it I was not specific:
There are two Vicon V962D-N312, both running the latest VCN01.2.1986.96 firmware.
Only one of them has the smart-motion problem.
Both use different models of WiFi bridges to connect, as the cameras are not natively WiFi enabled but are connected over WiFi. This makes me suspect the issue is not necessarily the particular video streams of the particular cameras, but issues in general when packets are lost or network congestion affects the streams momentarily and ephemerally.
Cheers
0 -
Hi Rick,
Without the options to investigate the issue, we only can guess what is going on.
It seems different than the Uniview issue, can you confirm that there is still a secondary stream available?
If there is, what is the resolution of the secondary stream?Is there a way to set up a similar camera on another location, maybe through your supplier of these cameras?
0 -
As I mentioned in the report, these cameras do not have working second stream, so motion is done on the primary stream .
I believe I can source a loaner camera, though everything is much slower in these days of pandemic isolation. If I can, I would configure it identically to the problematic cameras, although I do not have an identical WiFi adapter.
If your interest is strictly the video stream format, I could stand up the camera in a DMZ and provide you access, would that suffice?
0 -
...and as I mentioned, I would very much like to see some kind of alert/logging ability for loss of smart-motion. Can this be logged here, or would you prefer I make the suggestion over in the other channel?
0 -
Hi Rick,
We can try to replicate it with access to the camera, but can't guarantee it will succeed in this way.
Please share the login details (credentials, public IP address, ports) to <removed>.
Currently, there is nothing on the roadmap for alerting/logging of the loss of motion detection. I'm also not sure how you should create something that alerts you when there is motion but no motion detection.
0 -
OK, I will do so as soon as I have procured the camera.
It was my understanding from Boris' comment that the system has ceased detecting motion server-side because of time desynchronization. Surely such as situation is detectable. I suppose it's fair to say, though, that if detected, it would always be considered an error, always recovered from, and thus no alerting is necessary. That is fair enough.
0 -
Hi Rick,
I added the camera to our test server and will let you know.
0 -
Thanks for the update
It has taken typically a few weeks before motion search starts failing
With everything dark, I'm not sure how you'll know when motion search has failed? I can see about putting something in the field for the camera to see.
If we don't see any failures on your end after a few weeks, I can investigate putting the camera on a wireless bridge.
0 -
I hope that at some moments the lights are turned on and that should cause motion detection.
Since we do not have a tracking system in this community, feel free to remind me in this topic when you like to have an update from us. I will get noted when this post is updated.0 -
Yes, the lights will be turned on every now and then. I noticed watching the stream with VLC that even with the lights out, occasionally I saw full-frame flashes, which I did never see with nx, so that might be VLC trying some kind of error concealment.
I put on a light show for the camera now, let me know if you'd prefer dark lab with occasional lights on.
Cheers
0 -
Greetings
Perhaps because motion was being triggered so constantly with the "light show", or perhaps because network connectivity is different on my test bench, I have not seen subsequent issues on this camera. I trust you did not either on your end?
I also note there was a release note of fixes to smart motion processing in the monthly patches, and of course 4.1 is imminent. So for the moment, it looks like we can consider this resolved?
I will decommission the test camera shortly
Regards, and hope you all are doing well during these unfortunate times.
0 -
Hi Rick Sayre,
Thank you for reminding me to this thread. My apologies for not getting back at you about this issue.
You are correct, and I didn't notice any issues on my end. I wanted to report that back to you, but I couldn't find anymore where this request came from. I searched a fair bit, but couldn't find it anymore.
In the meantime, I have changed the workflow, and for each community request I create an internal ticket, so I always can find back the source.
I'm doing fine, I'm used to working from home and therefore the whole Covid-19 situation didn't affect me that much.
I hope you're doing well as well.0
Please sign in to leave a comment.
Comments
20 comments