Many errors while trying to use PTZ cameras
Completed
Hi all!
At current project we have a lot of issues with PTZ / speed-dome cameras: camera control become very "rude" and we have big delay on controlling scene.
We've found some errors in NX Witness log:
2021-08-03 15:21:20.095 103682 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.100 103679 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.120 103670 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.127 103734 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.130 103674 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.132 103735 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.150 103741 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.174 103748 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
2021-08-03 15:21:20.190 103753 WARNING QnOnvifPtzController(0x7f61f6c52770): Execution of PTZ status command for resource 'SD9362-EHL' has failed with error Fault info. Reason: Operation now in progress. Additional: SOAP-ENV:Receiver. Additional details: connect failed in tcp_connect(). .
Cameras:
- VIVOTEK SD9262-EHL
- Milesight MS-C2971-X23RPB
- AXIS P5534
Servers hive based on 3 virtual servers:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
NX Witness Server version:
$ cat /opt/networkoptix/mediaserver/build_info.txt
version=4.2.0.33306
fullVersion=4.2.0.33306-053a3f31dba7-default-patch
publicationType=patch
customization=default
cloudHost=nxvms.com
cloudGroup=prod
arch=x64
platform=linux
platformModification=ubuntu
changeSet=053a3f31dba7
branch=vms_4.2_patch
currentRefs=vms_4.2_patch,origin/vms_4.2_patch
brand=hdwitness
protoVersion=4201
metaVersion=
$ cat specific_features.txt
advanced_lens_control=1
mediaserver_metrics=1
get_time_of_servers_version=2
camera_auth_server_side_encryption=1
set_camera_param_post=1
primaryTimeServerDefinesInternetTimeSync=1
restartMethodVersion=2
layoutApiVersion=1
merge_history=1
vms_metrics=1
merge_systems=1
get_camera_param_manifest=1
-
According to the log you provided:
Additional details: connect failed in tcp_connect().
indicates the Nx Server could not establish or there are some issues with the TCP connection with the camera.
Please follow this support ARTICLE to capture and share the packets between the Nx Server and the camera with us for analysis.
Please capture the packets between the different cameras separately, make sure the packets you captured have covered the issue, and provide the IP address of each device, including the Nx Server and cameras.Thank you.
-
AFAIS, for doing PTZ actions NX client calls REST API handler /api/ptz
2021-08-07 23:20:57.748 3447589 DEBUG QnPtzRestHandler(0x7f9b9c68dd60): received request /api/ptz zSpeed=1;rotationSpeed=0;command=ContinuousMovePtzCommand;xSpeed=0;ySpeed=0;type=1;sequenceId={425489a9-50f1-436e-b264-b520965e9328};sequenceNumber=199;cameraId={6de9e378-4972-401d-c7d3-4087000ab547};
2021-08-07 23:20:57.749 3447591 DEBUG QnPtzRestHandler(0x7f9b9c68dd60): received request /api/ptz zSpeed=0;rotationSpeed=0;command=ContinuousMovePtzCommand;xSpeed=0;ySpeed=0;type=1;sequenceId={425489a9-50f1-436e-b264-b520965e9328};sequenceNumber=198;cameraId={6de9e378-4972-401d-c7d3-4087000ab547};And, for setting preset
2021-08-07 23:21:03.672 3447591 DEBUG QnPtzRestHandler(0x7f9b9c68dd60): received request /api/ptz command=GetPresetsPtzCommand;cameraId={6de9e378-4972-401d-c7d3-4087000ab547};
2021-08-07 23:21:03.673 3447628 DEBUG QnPtzRestHandler(0x7f9b9c68dd60): received request /api/ptz speed=1;command=ActivatePresetPtzCommand;presetId={43f4e5d2-1bcb-46cd-a509-7bdc25f22b4c};cameraId={6de9e378-4972-401d-c7d3-4087000ab547};Which version of HTTP protocol is using for requests between Client and Server?
Do we support Keep-Alive? Or connections between Client and Server always re-started?Since camera has limitations for opened TCP connections, server handle a lot of TCP connections (as I described at https://support.networkoptix.com/hc/en-us/community/posts/4405864721815-A-lot-of-TIME-WAIT-TCP-connections-when-video-walls-on), it may be the reason of issue with tcp_connect()
BTW, can we modify params for PTZ actions? Like, we need to speed-up Zoom and X or Y speed for production reasons (in browser - with native ActiveX plugin it works faster)
-
Another question:
How REST request routed on server side?
For example,
we have client, connected to server 1;
we have camera, added to server 2;
we sent command for move camera to left by clicking left arrow button in NX Witness client;
camera moves to the left and stops.
Expected routing:
REST call was directly to server 22021-08-07 23:21:03.672 3447591 DEBUG QnPtzRestHandler(0x7f9b9c68dd60): received request /api/ptz command=GetPresetsPtzCommand;cameraId={6de9e378-4972-401d-c7d3-4087000ab547};Actual routing:
?Why we need it?
Currently, in one of our project, we have scenario, where VMS operators using arrows for moving PTZ cameras.
We have 4 clients, that may control PTZ cameras in same time and each of them can click 3-4 times per second on arrows button in theirs clients.
Time after time, we have situations when we see "Moving..." text box on camera view and camera doesn't respond on any other commands.
Also, we have issues with delay, but network have tiny latencies.
What we should do to improve user experience?
Using mouse, or Advanced PTZ is not applicable, because of specific domain reasons. -
Hi Mykola,
Which version of HTTP protocol is using for requests between Client and Server?
Do we support Keep-Alive? Or connections between Client and Server always re-started?Generally GUI client reuses tcp connection for http requests.
Since camera has limitations for opened TCP connections, server handle a lot of TCP connections (as I described at https://support.networkoptix.com/hc/en-us/community/posts/4405864721815-A-lot-of-TIME-WAIT-TCP-connections-when-video-walls-on), it may be the reason of issue with tcp_connect()
Would it be possible to provide remote access to the server for additional troubleshooting? According to your log above, it looks tcp_connect() errors is triggered inside gSoap library, which is used by mediaserver. What about CPU/network load on the server? Do you have same errors on all 3 servers in the system?
BTW, can we modify params for PTZ actions? Like, we need to speed-up Zoom and X or Y speed for production reasons (in browser - with native ActiveX plugin it works faster)
Do you have this issue with one specific model/vendor of cameras? We need to take a closer look at the issue to see if it will be possible to fine-tune the sestitivty of the PTZ.
Expected routing:
REST call was directly to server 2This is what actually happens. Only server with PTZ camera can process PTZ api from GUI client.
We have 4 clients, that may control PTZ cameras in same time and each of them can click 3-4 times per second on arrows button in theirs clients.
Time after time, we have situations when we see "Moving..." text box on camera view and camera doesn't respond on any other commands.I did a few tests but couldn't reproduce this issue locally. Can this be reproduced without attempting to control 1 camera from 2 GUI clients at the same time?
Would it be possible to provide us with remote access to the system? We can convert this thread to support ticket so you can share access credentials securely.
-
Would it be possible to provide remote access to the server for additional troubleshooting?
Sorry, but for now we can't provide you direct remote access for additional troubleshooting for security reasons, but we can you your 'hand and eyes' and do whatever you ask.
What about CPU/network load on the server?
We have 10Gbit interfaces for servers, and only 300mbit/s load on it.
CPU - 40-45%, 55% is max.Do you have same errors on all 3 servers in the system?
Evenly, this issues appear on each of servers.
We have 3 video walls, and each of them connected to 1 of 3 of servers (https://support.networkoptix.com/hc/en-us/community/posts/4405864721815-A-lot-of-TIME-WAIT-TCP-connections-when-video-walls-on), and when we reach growth of TIME_WAIT connections for server, issues appear with more frequency.Do you have this issue with one specific model/vendor of cameras?
VIVOTEK SD9362-EHL
AXIS P5534
Milesight MS-C2971-X23RPBEach of them have those issues, but, AFAIS, AXIS have less one.
This is what actually happens. Only server with PTZ camera can process PTZ api from GUI client.
AFAIS, it is not correct (for 100%)
I've set up ssh tunnel between my client and 1 of 3 of servers.
On this side I have MacBook Pro M1 with newest NX Witness client.
So, I can manipulate PTZ from different servers.
AFAIU, it means my "entrypoint" routes API requests to other servers.Would it be possible to provide us with remote access to the system? We can convert this thread to support ticket so you can share access credentials securely.
I will try my best to make it real ASAP.
-
Just checking in to see whether you are able to provide remote access to the system for us to investigate the issue. Please let us know if you still need help by responding to this post. We are eager to resolve your question as soon as possible.
Thank you.
Please sign in to leave a comment.
Comments
6 comments