[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-09-07 Thread Alex Bligh
Note also that if vsftp continues to use clone(NEW_NETNS) (i.e. network namespaces) it is likely to suffer from #843892 anyway, so not using network namespaces will give you a stability increase. (NB - I have not tested vsftp against the bug in #843892 but as you can see from the text, it is hardly

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-09-07 Thread Alex Bligh
That is sadly not an option. LTS-backport kernel has a spectacular and easy to repeat Oops when namespaces are used. See #843892. It is not guaranteed by the kernel (it certainly wasn't in 2.6.32) that namespaces would be created and deleted instantly and without undue system pressure. It seems to

Re: [Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-09-07 Thread Stefan Bader
On 07.09.2011 12:16, Alex Bligh wrote: > The released resolution broke a production environment here: See #844185 > > I propose this is instead fixed by disabling it in vsftpd. > The problem is that nobody can say that vsftp was or is the only vector that allows to DOS a system doing something th

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-09-07 Thread Alex Bligh
The released resolution broke a production environment here: See #844185 I propose this is instead fixed by disabling it in vsftpd. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftp

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-06-03 Thread Stefan Metzmacher
As far as I understand the problem, the problem comes with creating a new network namespace with every clone() syscall. In a lxc setup only the startup process creates a new network namespace, just once. I can't see why vsftpd (without CLONE_NEWNET) won't run within an already established lxc ses

Re: [Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-06-02 Thread Rachel Greenham
On 02/06/11 14:32, Rachel Greenham wrote: > On 01/06/11 17:08, Stefan Metzmacher wrote: >> Fixing vsftpd looks like a much better fix for this Also presumably disabling it in vsftpd will hurt people who want to use that in an lxc setting without providing an easily-applied solution. -- Rachel

Re: [Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-06-02 Thread Rachel Greenham
On 01/06/11 17:08, Stefan Metzmacher wrote: > Fixing vsftpd looks like a much better fix for this It would seem at first sight to be simpler; but presumably the problem was that there are bugs in the implementation in the Lucid kernel (and upstream) that won't necessarily *only* impact vsftpd us

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-06-01 Thread Stefan Metzmacher
Fixing vsftpd looks like a much better fix for this! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a vmalloc space leak in Lucid -- ubuntu-bugs mailing list ubuntu-bugs

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-06-01 Thread Serge Hallyn
Actually, this isn't making sense to me. CLONE_NEWNET requires privilege, so this isn't something a random user can exploit. So what is the value in turning netns support off in the kernel as opposed to just stopping vsftpd from using it? (Attached debdiff not tested, but should suffice. I'll t

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-06-01 Thread Stefan Metzmacher
Yes, lxc is broken now see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/790863 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a vmalloc space leak in Lucid -- ub

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-05-31 Thread Serge Hallyn
lxc is now not usable on lucid. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a vmalloc space leak in Lucid -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com htt

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-05-29 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 2.6.32-32.62 --- linux (2.6.32-32.62) lucid-proposed; urgency=low [ Brad Figg ] * Release Tracking Bug - LP: #767370 [ Stefan Bader ] * (config) Disable CONFIG_NET_NS - LP: #720095 [ Upstream Kernel Changes ] * Revert

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-26 Thread Clint Byrum
** Tags added: verification-done ** Tags removed: verification-needed-lucid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a vmalloc space leak in Lucid -- ubuntu-bugs m

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-26 Thread Rachel Greenham
Applied successfully to two test instances; can confirm absence of netns process, and nothing seems to be broken. :-) My problem is only exhibiting on a production server under load though, and while I can see elevated netns cpu usage most of the time it only becomes a problem intermittently, so it

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-25 Thread Steve Conklin
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed' to 'verification-done'. See https://wiki.ubuntu.com/Testing/EnableProposed for documentatio

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-25 Thread Launchpad Bug Tracker
** Branch linked: lp:ubuntu/lucid-proposed/linux-ec2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a vmalloc space leak in Lucid -- ubuntu-bugs mailing list ubuntu-bugs

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-17 Thread Rachel Greenham
Quite right, I should have noticed that. :-) I may just need to be patient then, although this being a production machine experiencing real problems since go-live with users connecting in volume may preclude patience. I had been considering embedding a java FTP server into our application instead,

Re: [Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-16 Thread Clint Byrum
Excerpts from Rachel Greenham's message of Sat Apr 16 11:25:10 UTC 2011: > I think I've been experiencing this bug on a production vmware guest > server running Lucid with vsftp being connected to frequently by client > machines. > > The thing is, this bug shows as being "fix committed" - and the

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-04-16 Thread Rachel Greenham
I think I've been experiencing this bug on a production vmware guest server running Lucid with vsftp being connected to frequently by client machines. The thing is, this bug shows as being "fix committed" - and the implication I get from comment #18 is that the current production (ie: not backport

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-30 Thread Stefan Bader
** Description changed: + SRU justification: + + Impact: With activated network namespace (CONFIG_NET_NS) support it is + possible to create new namespaces much faster than cleaning them up. + This can lead to memory pressure and in the case of vsftp it is easily + possible to bring down the serv

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-30 Thread Stefan Bader
The problem only occurs on Lucid with network namespaces turned on. So not valid for Maverick and later. ** Changed in: linux (Ubuntu) Status: Confirmed => Invalid ** Changed in: linux (Ubuntu) Assignee: Stefan Bader (stefan-bader-canonical) => (unassigned) -- You received this bug

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-30 Thread Stefan Bader
After trying various approaches of backports which all seemed not really satisfying, it was decided that the safest way to go is to just turn off support for network namespaces. While this can have some impact on use- cases which try to containerize network, the feature was too immature to be turne

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-22 Thread Stefan Bader
Yes, basically cleanup is rather done in batches, which takes more cpu but could also affect lock contention. That and the fact that it requires backporting several patches which may cause effects we don't know of, causes me to be a bit reluctant about the changes. -- You received this bug notifi

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-21 Thread Jef Goupil
Hi, Sorry, I forgot to add a comment about the process netns: it was using a lot of CPU during my tests...: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 37 root 20 0

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-21 Thread Jef Goupil
Hi, I installed the following files from your site and rebooted: /root/linux-headers-2.6.32-31-server_2.6.32-31.60+netnsbp1_amd64.deb /root/linux-headers-2.6.32-31_2.6.32-31.60+netnsbp1_all.deb /root/linux-image-2.6.32-31-server_2.6.32-31.60+netnsbp1_amd64.deb /root/linux-libc-dev_2.6.32-31.60+ne

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-03-17 Thread Stefan Bader
Unfortunately this got a bit hit by other issues I was looking at. So I did not really see any small change to improve things. I guess I need to reach out for help from upstream. Meanwhile there is this series of patches I backported from 2.6.35 that allow network namespaces to be cleaned up in

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-25 Thread Stefan Bader
The privacy extension message was more of a side note. It has been removed in current code (probably because of the limited use). So the test case is making sense. And I was also beginning to think whether this could be seen as a security issue. As someone could bring down the server by doing many

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-24 Thread Jef Goupil
Hi, our production servers have more than 25 vsftpd connections for day: # zgrep CONNECT vsftpd.log.3.gz|wc -l 260210 which is about 3 connections per second. Each connection may have up to 200 file transferts. The testcase produces about 15 connections per seconds. It is 5 times more than

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-24 Thread Stefan Bader
So this is what I found out so far: whenever a client connects to vsftpd, it forks of a process to handle the connection. This is done in a way that also duplicates the network namespace (beside of the process namespace). This can actually be observed by the fact that every time that happens there

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-17 Thread Stefan Bader
Ok, so this is not really something conclusive yet, but it seems to me (when playing with that locally) that for that memory allocations to grow, there is no actual file transfer needed. Just looping through doing a connect and immediately disconnect again showed those 2M chunks growing. So I guess

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-16 Thread Peter Matulis
** Description changed: A simple stress test conducted on a KVM guest running standard updated Lucid with vsftpd demonstrates that memory is continuously used up until OOM Killer starts to protect the system (after ~12 min on my system). If test is terminated before that point is reached t

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-16 Thread Jef Goupil
Hi, I am working with Walter Richards. During our testing, we realized that it takes several hours to retrieve the memory after we stopped the ftp connections. On our 32GB Dell server, it took 5 hours to fill all the memory (until OOM kills at 2.1GB of free memory). Then, it took 8 hours before i

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-16 Thread Walter Richards
Hi, we have this issue on bare metal installation as you say. On a Dell M605 (AMD processor) and on an IBM x3650 (Intel). As well, it might help you to know that we tested it on SUSE with kernel 2.6.32-24, and the bug is not there. -- You received this bug notification because you are a member o

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-16 Thread Stefan Bader
** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Status: New => Confirmed ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Stefan Bader (stefan-bader-canonical) -- You received this bug notification because you are a member of Ubu

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-16 Thread Stefan Bader
It may be interesting if you had a spare disk and could do a bare metal installation of Lucid to repeat that test there. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a v

[Bug 720095] Re: vsftpd causes a vmalloc space leak in Lucid

2011-02-16 Thread Stefan Bader
Seems we got this already in Lucid. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/720095 Title: vsftpd causes a vmalloc space leak in Lucid -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com