Memory leak in 4.1.0.31398 on Ubuntu

Answered

Comments

12 comments

  • Avatar
    Norman - Nx Support

    Hi Michael Pasqualone,

    Can you clarify a bit more about what I see in the graphs? 

    When I think of a memory leak, I think about a creeping line to the top, while in your graphs I see eventually a shark tooth graph which isn't typical for a memory leak.  

    Also, the average memory usage of 2.34 GB doesn't sound really strange. That being said, obviously I have no clue about your system. Please provide the relevant information according to THIS support article. 

    --------------------

    Regarding your PS: 

    >>> P.S. I would submit a support ticket, but we're only able to submit a licensing request.
    >>> And please don't tell us to go to our licensing distributor, those guys have no idea what they're doing.

    That is in general what we recommend. Resellers can submit support tickets and for downstream customers we have this community. Reaching out to your reseller your provide you the fastest responses and if your reseller have no idea, you would help them and eventually yourself by keeping them in the middle since, you should get faster responses, and they learn by experience, so the next time they can help their customers themselves without our support.  

     

    0
    Comment actions Permalink
  • Avatar
    Michael Pasqualone

    Hey Norman,

    Our licensing distributor doesn't provide support. In this respect, we are the "reseller" - but unable to submit tickets. We used to be able to, so not sure what's going on there.

    The graphs are showing you the underlying server running out of memory, because the mediaserver-bin process is not returning memory back to the OS since upgrade to 4.1.0.31390.

    Here is a memleak log of the mediaserver-bin process id: https://gist.github.com/michaelp85/f4751fa29e80b844674650a227fbd01d

    Here's some extra information; this shows the vms server process gradually consuming more memory and at the highlighted part is where it finally crashed/restarted and thus memory usage reset back down to 12%. This will repeat each day.

    #/opt/networkoptix/mediaserver/var/log# cat log_file.log | grep memory

    2020-09-17 00:07:05.258 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 84.75%
    2020-09-17 00:07:05.258 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 77.40%
    2020-09-17 00:37:05.264 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 85.54%
    2020-09-17 00:37:05.264 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 78.05%
    2020-09-17 01:07:05.274 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 86.05%
    2020-09-17 01:07:05.274 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 78.47%
    2020-09-17 01:37:05.282 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 87.38%
    2020-09-17 01:37:05.282 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 79.73%
    2020-09-17 02:07:05.288 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 89.45%
    2020-09-17 02:07:05.288 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 81.74%
    2020-09-17 02:37:05.293 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 91.03%
    2020-09-17 02:37:05.293 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 83.26%
    2020-09-17 03:07:05.300 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 92.03%
    2020-09-17 03:07:05.300 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.53%
    2020-09-17 03:37:05.305 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 93.74%
    2020-09-17 03:37:05.305 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.97%
    2020-09-17 04:07:05.312 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 92.49%
    2020-09-17 04:07:05.312 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 83.11%
    2020-09-17 04:37:05.318 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 94.77%
    2020-09-17 04:37:05.318 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.86%
    2020-09-17 05:07:05.332 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.61%
    2020-09-17 05:07:05.332 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.51%
    2020-09-17 05:37:05.340 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.52%
    2020-09-17 05:37:05.340 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.26%
    2020-09-17 06:07:05.357 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.47%
    2020-09-17 06:07:05.357 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.11%
    2020-09-17 06:37:05.368 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 90.83%
    2020-09-17 06:37:05.368 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 79.56%
    2020-09-17 07:07:05.380 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 93.23%
    2020-09-17 07:07:05.380 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 82.13%
    2020-09-17 07:37:05.388 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 94.95%
    2020-09-17 07:37:05.388 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.50%
    2020-09-17 08:07:05.401 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 94.85%
    2020-09-17 08:07:05.401 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 83.42%
    2020-09-17 08:37:05.413 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.62%
    2020-09-17 08:37:05.413 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 84.81%
    2020-09-17 09:07:05.424 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 95.93%
    2020-09-17 09:07:05.424 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 85.13%
    2020-09-17 09:37:05.690 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): OS memory usage 97.04%
    2020-09-17 09:37:05.691 16779 INFO nx::vms::server::GlobalMonitor(0x7f505c036010): Process memory usage 86.00%
    2020-09-17 10:31:03.278 6411 INFO nx::vms::server::GlobalMonitor(0x7f90ec036010): OS memory usage 12.33%
    2020-09-17 10:31:03.278 6411 INFO nx::vms::server::GlobalMonitor(0x7f90ec036010): Process memory usage 9.69%

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Thank you Michael Pasqualone,

    Please upgrade to the latest patch with the following in-client upgrade credentials? 

    Build Number: 31767
    Password: l2eyxl

    Some improvements were made that might apply to your case. 

    If, for some reason, that isn't the case, we will move this request into a support ticket and will request access to an affected  server for further investigation with the aid of our developers.

     

    0
    Comment actions Permalink
  • Avatar
    Michael Pasqualone

    Hey Norman,

    We're 3 hours into running Build 31767, so far early signs are showing this is much more stable. Memory usage has stabilised now for ~2 hours at expected levels and isn't growing out of control.

    I won't call it just yet, this thing was crashing every 24 hours - so I'll report back in ~20 hours whether we've seen another build up to a crash or not, but so far signs are good.

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Thank you Michael Pasqualone,

    Looking forward to your update. 

    0
    Comment actions Permalink
  • Avatar
    Michael Pasqualone

    Hey Norman,

    We're still seeing issues. Not as extensively as before, but we've still got rolling outages/restarts and crashes.

    I think it's time to turn this into a support ticket. How do we go about that?

    Thanks,
    Michael

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Michael Pasqualone,

    I just pushed the button to move the request into a support ticket. 
    It might take a bit of time before we reply in the ticket, since we need to loop in our developers. 

    0
    Comment actions Permalink
  • Avatar
    Artem Zaytsev

    Hi Michael.

    Have you received a ticket update notification in which I requested an access to the system?

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Michael Pasqualone,

    Our team is awaiting your reply in the ticket. 
    Have you received our notification regarding the ticket? 

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Michael Pasqualone,

    Our team is awaiting your reply in the ticket. 
    Have you received our notification regarding the ticket?

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Michael Pasqualone,

    We have not heard back from you in a while, so we are going to assume your question has been answered and set this topic to 'Answered'.

    0
    Comment actions Permalink
  • Avatar
    Norman - Nx Support

    Hi Lieven Samijn,

    Please submit a ticket for further investigations. Thank you. 

    0
    Comment actions Permalink

Please sign in to leave a comment.