Memory leak in 4.1.0.31398 on Ubuntu
Answered
We're seeing memory leak issues with 4.1.0.31398 across all our Ubuntu VMs (Different ESXi hosts, different underlying hardware). We did not experience these issues on 4.0.0.29987.
root@nxw1-44pir:/opt/networkoptix/mediaserver/var/log# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
P.S. I would submit a support ticket, but we're only able to submit a licensing request. Our licensing distributor doesn't provide us support.
Here is our monitoring showing that the memory leaks started when we upgraded to to 4.1. The large change in available memory is us adding more memory to the VM as a test.
[QDateTime(2020-09-11 20:17:58.804 ACST Qt::TimeSpec(LocalTime))] Starting system update:
Current version: 4.0.0.29987
Target version: 4.1.0.31398
-
Can you clarify a bit more about what I see in the graphs?
When I think of a memory leak, I think about a creeping line to the top, while in your graphs I see eventually a shark tooth graph which isn't typical for a memory leak.
Also, the average memory usage of 2.34 GB doesn't sound really strange. That being said, obviously I have no clue about your system. Please provide the relevant information according to THIS support article.
--------------------
Regarding your PS:
>>> P.S. I would submit a support ticket, but we're only able to submit a licensing request.
>>> And please don't tell us to go to our licensing distributor, those guys have no idea what they're doing.That is in general what we recommend. Resellers can submit support tickets and for downstream customers we have this community. Reaching out to your reseller your provide you the fastest responses and if your reseller have no idea, you would help them and eventually yourself by keeping them in the middle since, you should get faster responses, and they learn by experience, so the next time they can help their customers themselves without our support.
-
Hey Norman,
Our licensing distributor doesn't provide support. In this respect, we are the "reseller" - but unable to submit tickets. We used to be able to, so not sure what's going on there.
The graphs are showing you the underlying server running out of memory, because the mediaserver-bin process is not returning memory back to the OS since upgrade to 4.1.0.31390.
Here is a memleak log of the mediaserver-bin process id: https://gist.github.com/michaelp85/f4751fa29e80b844674650a227fbd01d
Here's some extra information; this shows the vms server process gradually consuming more memory and at the highlighted part is where it finally crashed/restarted and thus memory usage reset back down to 12%. This will repeat each day.
#/opt/networkoptix/mediaserver/var/log# cat log_file.log | grep memory
2020-09-17 00:07:05.258 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 84.75%
2020-09-17 00:07:05.258 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 77.40%
2020-09-17 00:37:05.264 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 85.54%
2020-09-17 00:37:05.264 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 78.05%
2020-09-17 01:07:05.274 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 86.05%
2020-09-17 01:07:05.274 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 78.47%
2020-09-17 01:37:05.282 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 87.38%
2020-09-17 01:37:05.282 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 79.73%
2020-09-17 02:07:05.288 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 89.45%
2020-09-17 02:07:05.288 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 81.74%
2020-09-17 02:37:05.293 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 91.03%
2020-09-17 02:37:05.293 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 83.26%
2020-09-17 03:07:05.300 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 92.03%
2020-09-17 03:07:05.300 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.53%
2020-09-17 03:37:05.305 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 93.74%
2020-09-17 03:37:05.305 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.97%
2020-09-17 04:07:05.312 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 92.49%
2020-09-17 04:07:05.312 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 83.11%
2020-09-17 04:37:05.318 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 94.77%
2020-09-17 04:37:05.318 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.86%
2020-09-17 05:07:05.332 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.61%
2020-09-17 05:07:05.332 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.51%
2020-09-17 05:37:05.340 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.52%
2020-09-17 05:37:05.340 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.26%
2020-09-17 06:07:05.357 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.47%
2020-09-17 06:07:05.357 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.11%
2020-09-17 06:37:05.368 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 90.83%
2020-09-17 06:37:05.368 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 79.56%
2020-09-17 07:07:05.380 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 93.23%
2020-09-17 07:07:05.380 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 82.13%
2020-09-17 07:37:05.388 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 94.95%
2020-09-17 07:37:05.388 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.50%
2020-09-17 08:07:05.401 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 94.85%
2020-09-17 08:07:05.401 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 83.42%
2020-09-17 08:37:05.413 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.62%
2020-09-17 08:37:05.413 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.81%
2020-09-17 09:07:05.424 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.93%
2020-09-17 09:07:05.424 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.13%
2020-09-17 09:37:05.690 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 97.04%
2020-09-17 09:37:05.691 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 86.00%
2020-09-17 10:31:03.278 6411 INFO nx::vms::server::GlobalMonitor(0x7f90ec036010): OS memory usage 12.33%
2020-09-17 10:31:03.278 6411 INFO nx::vms::server::GlobalMonitor(0x7f90ec036010): Process memory usage 9.69% -
Thank you Michael Pasqualone,
Please upgrade to the latest patch with the following in-client upgrade credentials?
Build Number: 31767
Password: l2eyxlSome improvements were made that might apply to your case.
If, for some reason, that isn't the case, we will move this request into a support ticket and will request access to an affected server for further investigation with the aid of our developers.
-
Hey Norman,
We're 3 hours into running Build 31767, so far early signs are showing this is much more stable. Memory usage has stabilised now for ~2 hours at expected levels and isn't growing out of control.
I won't call it just yet, this thing was crashing every 24 hours - so I'll report back in ~20 hours whether we've seen another build up to a crash or not, but so far signs are good.
-
Thank you Michael Pasqualone,
Looking forward to your update.
-
I just pushed the button to move the request into a support ticket.
It might take a bit of time before we reply in the ticket, since we need to loop in our developers. -
Our team is awaiting your reply in the ticket.
Have you received our notification regarding the ticket? -
Our team is awaiting your reply in the ticket.
Have you received our notification regarding the ticket? -
We have not heard back from you in a while, so we are going to assume your question has been answered and set this topic to 'Answered'.
-
Hi Lieven Samijn,
Please submit a ticket for further investigations. Thank you.
Please sign in to leave a comment.
Comments
12 comments