API Read Timeouts
Completed
Hi,
We have a server VMS hive with 20 servers. I'm pulling bookmarks via the REST API and it works well and is responsive. From time to time the server runs a re-index and a scan on the storage and when this happens the API pulling the bookmarks returns a timeout and does not respond at all, after a while the system recovers and the API is responsive again.
Additionally, I can see in my logs that the API starts timing out every day between 6am and 7am in the morning and after a few minutes recovers.
I don't see any messages in the system logs. Is there any way that we can pinpoint why the API is timing out so we can address the issue?
-
Hi Reynard,
could you please provide more info on your environment?
do you have hive servers located locally or all of them are speared in different networks? Do you send your API request to specific server with the camera or just random server? Does the issue occur for bookmarks located on all servers or only bookmarks from specific server can't be accessed?
First of all, I'd recommend to run test where you'd be requesting bookmarks directly from the server that has them stored. This way it will be possible to exclude hive networking and locate the server with an issue.
Can you tell which VMS build are you using?
-
Hi Anton,
Our servers are located in our datacenter rack. We joined a few failover servers that is on a separate network - the 2 locations are linked via layer 2 E-Tree.
I'm sending the API request to a single server with multiple camera IDs spread across the servers in the hive.
We are running V5.0.0.34971 - we recently upgraded. Before the upgrade I could see that if a VMS started to re-index and do a storage scan, the bookmarks for that VMS would be pulled when the storage scan is completed. After the upgrade it seems like the API times out (what I'm guessing). Because I'm pulling multiple cameras bookmarks, I cant pinpoint why the API times out at times. Sometimes it happens without a VMS doing a storage scan.
I will try to pull the camera information from the VMS it is loaded on and see how the API performs.
-
Hello Reynard Bubb,
Thank you for explanation.
Could you give more details on your system, please?
Number of servers, number of cameras on each server, server performance screenshots while you're requesting bookmarks, particular API request you're executing.
After the upgrade it seems like the API times out
Have you managed to assess the duration in which it "times out"?
Could you please join to the hive a server with no cameras connected and try to send requests to it? Does the issue persist?
the bookmarks for that VMS would be pulled when the storage scan is completed
The storage rescans is not a standard situation. During the stable functioning of the Server, they don't happen. They are started manually or started automatically when .nxdb file is detected to be broken.
I'd suggest resolving this issue first.
Please sign in to leave a comment.
Comments
3 comments