Dear Danny,
the following message means that it was not global memory shortage,
[ 841.805656] 6716 (mysqld) invoked oom-killer in ub 0 generation 0 gfp 0x200d2
It is caused by per-ct memory management.|
However "ub 0" means that memory shortage was happen not in real container but
in so-called VE0, i.e. on host itself.
It looks like some misconfiguration for me.
I expect you have assigned some strong memory limits for VE0,
and OOM-killer was executed when host processes overuse this limit.
Please check do you probably have /etc/vz/conf/0.conf file on affected node?
Thank you,
Vasily Averin
On 2017-09-29 09:14, Danny Gurman wrote:
> Hello Vasily ,
>
> First - thanks for your ultra-fast response :-)
>
> Note that for debugging purpose the container (there is a single one) was
> manually
> stopped (vzctl stop 101)immediately as kernel started.
>
> I Just want to clarify that we did not observe global memory shortage on the
> host (see attached "free" command output and meminfo).
> We first notice the issue on 24 RAM and then we increased physical RAM to 32
> - exactly same results.
> The issue happen always when RAM usage get to 13 GB.
>
> Regarding forcing kernel for to get memory strongly from specified memory
> zone - not familiar with those setting.
> Is there any command to check this ?
>
> - Note also that machine was deployed on VMWare.
>
> I am attaching:
> 1. dmesg output.
> 2. sysctl -a output
> 3. Output of script that run every minute and output "free" and meminfo (the
> issue reproduced in ~ 10 minutes so it's not long).
> 4. Short version of the previous - the state before OOM killer invocation.
>
> Thanks,
> Danny
>
>
>
> -----Original Message-----
> From: Vasily Averin [mailto:v...@virtuozzo.com]
> Sent: Friday, September 29, 2017 7:46 AM
> To: OpenVZ users <users@openvz.org>; Danny Gurman <dan...@radware.com>
> Cc: Liat Vaknin <li...@radware.com>; Dima Paikin <di...@radware.com>; Lior
> Komanski <li...@radware.com>; Yohai Liebman <yoh...@radware.com>; Rotem Inbar
> <rot...@radware.com>
> Subject: Re: [Users] OOM Killer invoked even that are available RAM and swap
> on the machine
>
> Dear Danny,
>
> 1) Are you sure that it was global memory shortage?
>
> On Virtuozzo kernels there is per-container OOM-killer, it is executed in
> case memory shortage inside affected container and it kills tasks related to
> this container only.
> It is normal behaviour,and you should not worry about such cases.
>
> Please read related kernel messages, it explain the reason of OOM.
>
> 2) if it was global OOM -- need to clarify in which memory zone it was happen.
>
> as you probably know memory in linux kernels is divided into few memory zones.
> Global OOM can happen when kernel was forced to get memory strongly from
> specified memory zone.
> The memory zone can have no pages that can be moved to swap, and when memory
> management cannot free memory in this zone by other ways it can call
> OOM-killer.
> This can happen, but it is quite rare case. Need to find how who had
> submitted such memory request.
>
> Frankly speaking I think you observe first of described scenarious.
>
> Thank you,
> Vasily Averin
>
> On 2017-09-29 02:17, Danny Gurman wrote:
>>
>>
>> Hello all,
>>
>> I will really appreciate assistance regarding the following issue:
>>
>>
>>
>> We have a product deployed on CentosOS 6.9 with OpenVZ.
>>
>> Machine RAM size – *32* GB + Swap file (16 GB).
>>
>> The following described issue was reproduced on the OpenVZ kernels:
>>
>> 2.6.32-042stab125.1 , 2.6.32-042stab123.9 , 2.6.32-042stab123.2,
>> 2.6.32-042stab120.11
>>
>>
>>
>> For debugging purpose the single *container* on the host*was stopped*.
>>
>>
>>
>> We observed that OOM Killer is invoked whenever host RAM usage without
>> buffers reach ~13 GB:
>>
>> Free command output (few seconds before OOM killer invoked):
>>
>> total used free shared buffers cached
>>
>> Mem: 32793080 23197272 9595808 920 135992 10276712
>>
>> -/+ buffers/cache: *12784568* 20008512
>>
>> Swap: 16465916 *0* 16465916
>>
>>
>>
>> On the message buffer, we see that the machine memory state just before the
>> OOM killer invocation is:
>>
>> RAM: *2097074* / 2097152 [1] SWAP: *1048576* / *1048576* [1] KMEM:
>> 227942400 /
>>
>>
>>
>> Again , machine has 32 GB, only 13 GB is used, no swap usage – so
>> where those wrong values came
>>
>> from and why OOM killer is invoked in this stage?
>>
>>
>>
>> The issue is reproducible with similar results every time.
>>
>>
>>
>> Is it a known issue ?
>>
>> Should I open a defect for this?
>>
>> Also – is there a way to totally disable the OOM killer?
>>
>>
>>
>> I’ll be glad to provide any required detail (like sysctl –a output)
>>
>>
>>
>> Regards,
>>
>> Danny
>>
>>
>>
>> Radware
>>
>>
>>
>> Danny Gurman
>> Dev Lead, APSolute Vision
>> Email: danny.gur...@gmail.com <mailto:danny.gur...@gmail.com>
>>
>>
>>
>> T:+972 723917088
>> M:+972 526111008
>> F:
>>
>>
>>
>>
>>
>> Check out the latest and greatest from Radware
>> <https://www.radware.com/Resources/CampaignRedirector.html>
>>
>>
>>
>> *www.radware.com <https://www.radware.com>*
>>
>>
>>
>>
>>
>>
>>
>> Blog <https://blog.radware.com/>
>> https://www.radware.com/images/signature/linkedin.jpg
>> <https://www.linkedin.com/companies/165642>https://www.radware.com/ima
>> ges/signature/twitter.jpg <file://twitter.com/radware> youtube
>> <https://www.youtube.com/user/radwareinc>
>>
>> Confidentiality note: This message, and any attachments to it, contains
>> privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not
>> be disclosed, used, copied, or transmitted in any form or by any means
>> without prior written permission from RADWARE. If you are not the intended
>> recipient, delete the message and any attachments from your system without
>> reading or copying it, and kindly notify the sender by e-mail. Thank you.
>>
>> *P****Please consider your environmental responsibility before
>> printing this e-mail***
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users@openvz.org
>> https://lists.openvz.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users
