Re: memory-cgroup bug

On Mon 26-11-12 01:38:55, azurIt wrote: > >This is hackish but it should help you in this case. Kamezawa, what do > >you think about that? Should we generalize this and prepare something > >like mem_cgroup_cache_charge_locked which would add __GFP_NORETRY > >automatically and use the function whene

Re: memory-cgroup bug

>This is hackish but it should help you in this case. Kamezawa, what do >you think about that? Should we generalize this and prepare something >like mem_cgroup_cache_charge_locked which would add __GFP_NORETRY >automatically and use the function whenever we are in a locked context? >To be honest I

Re: memory-cgroup bug

On Sun 25-11-12 13:05:24, Michal Hocko wrote: > [Adding Kamezawa into CC] > > On Sun 25-11-12 01:10:47, azurIt wrote: > > >Could you take few snapshots over time? > > > > > > Here it is, now from different server, snapshot was taken every second > > for 10 minutes (hope it's enough): > > www.wat

Re: memory-cgroup bug

On Sun 25-11-12 14:27:09, azurIt wrote: > >> Thank you very much, i will install it ASAP (probably this night). > > > >Please don't. If my analysis is correct which I am almost 100% sure it > >is then it would cause excessive logging. I am sorry I cannot come up > >with something else in the mean t

Re: memory-cgroup bug

>> Thank you very much, i will install it ASAP (probably this night). > >Please don't. If my analysis is correct which I am almost 100% sure it >is then it would cause excessive logging. I am sorry I cannot come up >with something else in the mean time. Ok then. I will, meanwhile, try to contact

Re: memory-cgroup bug

On Sun 25-11-12 13:39:53, azurIt wrote: > >Inlined at the end of the email. Please note I have compile tested > >it. It might produce a lot of output. > > > Thank you very much, i will install it ASAP (probably this night). Please don't. If my analysis is correct which I am almost 100% sure it i

Re: memory-cgroup bug

>Inlined at the end of the email. Please note I have compile tested >it. It might produce a lot of output. Thank you very much, i will install it ASAP (probably this night). >dmesg | grep "Out of memory" >doesn't tell anything, right? Only messages for other cgroups but not for the freezed on

Re: memory-cgroup bug

>So there is a lot of attempts to allocate which fail, every second! Yes, as i said, the cgroup was taking 100% of (allocated) CPU core(s). Not sure if all processes were using CPU but _few_ of them (not only one) for sure. -- To unsubscribe from this list: send the line "unsubscribe linux-kerne

Re: memory-cgroup bug

[Adding Kamezawa into CC] On Sun 25-11-12 01:10:47, azurIt wrote: > >Could you take few snapshots over time? > > > Here it is, now from different server, snapshot was taken every second > for 10 minutes (hope it's enough): > www.watchdog.sk/lkml/memcg-bug-2.tar.gz Hmm, interesting: $ grep . */m

Re: memory-cgroup bug

On Fri 23-11-12 15:59:04, azurIt wrote: > >If you could instrument mem_cgroup_handle_oom with some printks (before > >we take the memcg_oom_lock, before we schedule and into > >mem_cgroup_out_of_memory) > > > If you send me patch i can do it. I'm, unfortunately, not able to code it. Inlined at t

Re: memory-cgroup bug

>Could you take few snapshots over time? Here it is, now from different server, snapshot was taken every second for 10 minutes (hope it's enough): www.watchdog.sk/lkml/memcg-bug-2.tar.gz -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord

Re: memory-cgroup bug

>If you could instrument mem_cgroup_handle_oom with some printks (before >we take the memcg_oom_lock, before we schedule and into >mem_cgroup_out_of_memory) If you send me patch i can do it. I'm, unfortunately, not able to code it. >> It, luckily, happend again so i have more info. >> >> - t

Re: memory-cgroup bug

On Fri 23-11-12 10:44:23, azurIt wrote: [...] > # gdb vmlinux > GNU gdb (GDB) 7.0.1-debian > Copyright (C) 2009 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > Ther

Re: memory-cgroup bug

On Fri 23-11-12 10:21:37, azurIt wrote: [...] > It, luckily, happend again so i have more info. > > - there wasn't any logs in kernel from OOM for that cgroup > - there were 16 processes in cgroup > - processes in cgroup were taking togather 100% of CPU (it >was allowed to use only one core

Re: memory-cgroup bug

> CC: linux-kernel@vger.kernel.org, linux...@kvack.org, "cgroups mailinglist" > >On Fri 23-11-12 10:21:37, azurIt wrote: >> >Either use gdb YOUR_VMLINUX and disassemble mem_cgroup_handle_oom or >> >use objdump -d YOUR_VMLINUX and copy out only mem_cgroup_handle_oom >> >function. >> If 'YOUR_VMLIN

Re: memory-cgroup bug

On 11/23/2012 01:21 PM, azurIt wrote: >> Either use gdb YOUR_VMLINUX and disassemble mem_cgroup_handle_oom or >> use objdump -d YOUR_VMLINUX and copy out only mem_cgroup_handle_oom >> function. > If 'YOUR_VMLINUX' is supposed to be my kernel image: > > # gdb vmlinuz-3.2.34-grsec-1 this is vmlinu

Re: memory-cgroup bug

On Fri 23-11-12 10:21:37, azurIt wrote: > >Either use gdb YOUR_VMLINUX and disassemble mem_cgroup_handle_oom or > >use objdump -d YOUR_VMLINUX and copy out only mem_cgroup_handle_oom > >function. > If 'YOUR_VMLINUX' is supposed to be my kernel image: > > # gdb vmlinuz-3.2.34-grsec-1 > GNU gdb (GD

Re: memory-cgroup bug

>Either use gdb YOUR_VMLINUX and disassemble mem_cgroup_handle_oom or >use objdump -d YOUR_VMLINUX and copy out only mem_cgroup_handle_oom >function. If 'YOUR_VMLINUX' is supposed to be my kernel image: # gdb vmlinuz-3.2.34-grsec-1 GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Found

Re: memory-cgroup bug

On Thu 22-11-12 23:34:34, azurIt wrote: [...] > >And finally could you post the disassembly of your version of > >mem_cgroup_handle_oom, please? > > How can i do this? Either use gdb YOUR_VMLINUX and disassemble mem_cgroup_handle_oom or use objdump -d YOUR_VMLINUX and copy out only mem_cgroup_han

Re: memory-cgroup bug

__ > Od: "Kamezawa Hiroyuki" > Komu: azurIt > Dátum: 22.11.2012 01:27 > Predmet: Re: memory-cgroup bug > > CC: linux-kernel@vger.kernel.org, "linux-mm" >(2012/11/22 4:02), azurIt wrote: >>

Re: memory-cgroup bug

>Btw. is this stack stable or is the task bouncing in some loop? Not sure, will check it next time. >And finally could you post the disassembly of your version of >mem_cgroup_handle_oom, please? How can i do this? >What does your kernel log says while this is happening. Are there any >mem

Re: memory-cgroup bug

On Thu 22-11-12 10:36:18, azurIt wrote: [...] > I can look also to the data of 'freezed' proces if you need it but i > will have to wait until problem occurs again. > > The main problem is that when this problem happens, it's NOT resolved > automatically by kernel/OOM and user of cgroup, where it

Re: memory-cgroup bug

On Thu 22-11-12 19:05:26, azurIt wrote: [...] > My cgroups hierarchy: > /cgroups//uid/ > > where '' is system user id and 'uid' is just word 'uid'. > > Memory limits are set in /cgroups// and hierarchy is > enabled. Processes are inside /cgroups//uid/ . I'm using > hard limits for memory and swap

Re: memory-cgroup bug

On Wed 21-11-12 20:02:07, azurIt wrote: > Hi, > > i'm using memory cgroup for limiting our users and having a really > strange problem when a cgroup gets out of its memory limit. It's very > strange because it happens only sometimes (about once per week on > random user), out of memory is usually

Re: memory-cgroup bug

(2012/11/22 4:02), azurIt wrote: Hi, i'm using memory cgroup for limiting our users and having a really strange problem when a cgroup gets out of its memory limit. It's very strange because it happens only sometimes (about once per week on random user), out of memory is usually handled ok. Th

Re: memory-cgroup bug

>> i'm using memory cgroup for limiting our users and having a really >> strange problem when a cgroup gets out of its memory limit. It's very >> strange because it happens only sometimes (about once per week on >> random user), out of memory is usually handled ok. > >What is your memcg configurati