it's actually 1st time I see limits for _host_ itself, is it somehow default installation or you modified it for some reason?
On Fri, Sep 29, 2017 at 7:27 PM, Danny Gurman <dan...@radware.com> wrote: > Vasily - you are the man > you were correct ! :-) > > > Setting /etc/vz/conf/0.conf with > "PHYSPAGES="0:unlimited" > SWAPPAGES="0:unlimited" > > Indeed solve the issue :-) > > Thanks so much for your assistant and effort, > Danny > > > -----Original Message----- > From: Vasily Averin [mailto:v...@virtuozzo.com] > Sent: Friday, September 29, 2017 2:03 PM > To: Danny Gurman <dan...@radware.com> > Subject: Re: [Users] OOM Killer invoked even that are available RAM and swap > on the machine > > On 2017-09-29 13:32, Danny Gurman wrote: >> Thanks again Vasily, >> >> Here are the values from /etc/vz/conf/0.conf: >> # UBC parameters (in form of barrier:limit) >> OOMGUARPAGES="unlimited:unlimited" >> CPUS="4" >> CPUMASK="0-3" >> #CPUUNITS="400000" >> PHYSPAGES="0:8G" >> SWAPPAGES="0:4G" >> >> So , what the physical pages limitation on VE0 (host) actually mean ? >> Host is limited to 8 GB RAM? >> Note that we reach ~ 13 GB. > > yes, I was wrong in previous letter, > I forget that PAGES paraments are showed in 4kb pages so RAM = 2097152 > means not 2 Gb but 8Gb, and SWAP = 1048576 means 4Gb virtual swap.. > so it correspond to your config. > > These parameters limits host processes by the same way as containers. > It means that host processes cannot use more that 12 Gb accounted memory, if > they overuse this limit -- OOM-killer will be called to decrease the usage. > > I would note -- it isn't default settings, so you can safely remove it. > > >> Of course we could try to change to unlimited:unlimited for check if it >> solve the issue. >> I'll update >> >> Regards, >> Danny >> >> -----Original Message----- >> From: Vasily Averin [mailto:v...@virtuozzo.com] >> Sent: Friday, September 29, 2017 12:29 PM >> To: Danny Gurman <dan...@radware.com>; OpenVZ users <users@openvz.org> >> Cc: Liat Vaknin <li...@radware.com>; Dima Paikin <di...@radware.com>; >> Lior Komanski <li...@radware.com>; Yohai Liebman <yoh...@radware.com>; >> Rotem Inbar <rot...@radware.com>; Alexander Lurye >> <alexand...@3vium.com>; Gal Yahel <g...@radware.com> >> Subject: Re: [Users] OOM Killer invoked even that are available RAM >> and swap on the machine >> >> Dear Danny, >> the following message means that it was not global memory shortage, [ >> 841.805656] 6716 (mysqld) invoked oom-killer in ub 0 generation 0 gfp >> 0x200d2 >> >> It is caused by per-ct memory management.| However "ub 0" means that memory >> shortage was happen not in real container but in so-called VE0, i.e. on host >> itself. >> It looks like some misconfiguration for me. >> I expect you have assigned some strong memory limits for VE0, and OOM-killer >> was executed when host processes overuse this limit. >> >> Please check do you probably have /etc/vz/conf/0.conf file on affected node? >> >> Thank you, >> Vasily Averin >> >> On 2017-09-29 09:14, Danny Gurman wrote: >>> Hello Vasily , >>> >>> First - thanks for your ultra-fast response :-) >>> >>> Note that for debugging purpose the container (there is a single >>> one) was manually stopped (vzctl stop 101)immediately as kernel started. >>> >>> I Just want to clarify that we did not observe global memory shortage on >>> the host (see attached "free" command output and meminfo). >>> We first notice the issue on 24 RAM and then we increased physical RAM to >>> 32 - exactly same results. >>> The issue happen always when RAM usage get to 13 GB. >>> >>> Regarding forcing kernel for to get memory strongly from specified memory >>> zone - not familiar with those setting. >>> Is there any command to check this ? >>> >>> - Note also that machine was deployed on VMWare. >>> >>> I am attaching: >>> 1. dmesg output. >>> 2. sysctl -a output >>> 3. Output of script that run every minute and output "free" and meminfo >>> (the issue reproduced in ~ 10 minutes so it's not long). >>> 4. Short version of the previous - the state before OOM killer invocation. >>> >>> Thanks, >>> Danny >>> >>> >>> >>> -----Original Message----- >>> From: Vasily Averin [mailto:v...@virtuozzo.com] >>> Sent: Friday, September 29, 2017 7:46 AM >>> To: OpenVZ users <users@openvz.org>; Danny Gurman >>> <dan...@radware.com> >>> Cc: Liat Vaknin <li...@radware.com>; Dima Paikin <di...@radware.com>; >>> Lior Komanski <li...@radware.com>; Yohai Liebman >>> <yoh...@radware.com>; Rotem Inbar <rot...@radware.com> >>> Subject: Re: [Users] OOM Killer invoked even that are available RAM >>> and swap on the machine >>> >>> Dear Danny, >>> >>> 1) Are you sure that it was global memory shortage? >>> >>> On Virtuozzo kernels there is per-container OOM-killer, it is executed in >>> case memory shortage inside affected container and it kills tasks related >>> to this container only. >>> It is normal behaviour,and you should not worry about such cases. >>> >>> Please read related kernel messages, it explain the reason of OOM. >>> >>> 2) if it was global OOM -- need to clarify in which memory zone it was >>> happen. >>> >>> as you probably know memory in linux kernels is divided into few memory >>> zones. >>> Global OOM can happen when kernel was forced to get memory strongly from >>> specified memory zone. >>> The memory zone can have no pages that can be moved to swap, and when >>> memory management cannot free memory in this zone by other ways it can call >>> OOM-killer. >>> This can happen, but it is quite rare case. Need to find how who had >>> submitted such memory request. >>> >>> Frankly speaking I think you observe first of described scenarious. >>> >>> Thank you, >>> Vasily Averin >>> >>> On 2017-09-29 02:17, Danny Gurman wrote: >>>> >>>> >>>> Hello all, >>>> >>>> I will really appreciate assistance regarding the following issue: >>>> >>>> >>>> >>>> We have a product deployed on CentosOS 6.9 with OpenVZ. >>>> >>>> Machine RAM size – *32* GB + Swap file (16 GB). >>>> >>>> The following described issue was reproduced on the OpenVZ kernels: >>>> >>>> 2.6.32-042stab125.1 , 2.6.32-042stab123.9 , 2.6.32-042stab123.2, >>>> 2.6.32-042stab120.11 >>>> >>>> >>>> >>>> For debugging purpose the single *container* on the host*was stopped*. >>>> >>>> >>>> >>>> We observed that OOM Killer is invoked whenever host RAM usage without >>>> buffers reach ~13 GB: >>>> >>>> Free command output (few seconds before OOM killer invoked): >>>> >>>> total used free shared buffers cached >>>> >>>> Mem: 32793080 23197272 9595808 920 135992 10276712 >>>> >>>> -/+ buffers/cache: *12784568* 20008512 >>>> >>>> Swap: 16465916 *0* 16465916 >>>> >>>> >>>> >>>> On the message buffer, we see that the machine memory state just before >>>> the OOM killer invocation is: >>>> >>>> RAM: *2097074* / 2097152 [1] SWAP: *1048576* / *1048576* [1] KMEM: >>>> 227942400 / >>>> >>>> >>>> >>>> Again , machine has 32 GB, only 13 GB is used, no swap usage – so >>>> where those wrong values came >>>> >>>> from and why OOM killer is invoked in this stage? >>>> >>>> >>>> >>>> The issue is reproducible with similar results every time. >>>> >>>> >>>> >>>> Is it a known issue ? >>>> >>>> Should I open a defect for this? >>>> >>>> Also – is there a way to totally disable the OOM killer? >>>> >>>> >>>> >>>> I’ll be glad to provide any required detail (like sysctl –a output) >>>> >>>> >>>> >>>> Regards, >>>> >>>> Danny >>>> >>>> >>>> >>>> Radware >>>> >>>> >>>> >>>> Danny Gurman >>>> Dev Lead, APSolute Vision >>>> Email: danny.gur...@gmail.com <mailto:danny.gur...@gmail.com> >>>> >>>> >>>> >>>> T:+972 723917088 >>>> M:+972 526111008 >>>> F: >>>> >>>> >>>> >>>> >>>> >>>> Check out the latest and greatest from Radware >>>> <https://www.radware.com/Resources/CampaignRedirector.html> >>>> >>>> >>>> >>>> *www.radware.com <https://www.radware.com>* >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Blog <https://blog.radware.com/> >>>> https://www.radware.com/images/signature/linkedin.jpg >>>> <https://www.linkedin.com/companies/165642>https://www.radware.com/i >>>> m a ges/signature/twitter.jpg <file://twitter.com/radware> youtube >>>> <https://www.youtube.com/user/radwareinc> >>>> >>>> Confidentiality note: This message, and any attachments to it, contains >>>> privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may >>>> not be disclosed, used, copied, or transmitted in any form or by any means >>>> without prior written permission from RADWARE. If you are not the intended >>>> recipient, delete the message and any attachments from your system without >>>> reading or copying it, and kindly notify the sender by e-mail. Thank you. >>>> >>>> *P****Please consider your environmental responsibility before >>>> printing this e-mail*** >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@openvz.org >>>> https://lists.openvz.org/mailman/listinfo/users >>> > > _______________________________________________ > Users mailing list > Users@openvz.org > https://lists.openvz.org/mailman/listinfo/users -- Best regards, [COOLCOLD-RIPN] _______________________________________________ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
