Dear Gert, Brad all, as I also had reported to have recognized, that memory was not freed, even
though the openvpn disconnect-script was disconnecting all of the 3400 Openvpn-sessions (pls see mail below). Would you think, this is also improved by the change from malloc to calloc, so we could update to 2.3.3 to improve the memory-leak situation? freundlich grüsst, kind regards, _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Arno Odermatt Phone +41414454053 arno.oderm...@ch.schindler.com Schindler Elevator Ltd. | Corporate Research & Development | CRD/CO-BAT Zugerstr. 13 | CH-6030 Ebikon, Switzerland http://www.schindler.com _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Please consider your environment. Schindler supports sustainable urban development with safe, reliable and ecologically sound mobility solutions. From: Gert Doering <g...@greenie.muc.de> To: Brad Zhang <hebei5...@gmail.com>, Cc: openvpn-devel@lists.sourceforge.net List-Post: openvpn-devel@lists.sourceforge.net Date: 20.11.2013 19:33 Subject: Re: [Openvpn-devel] Does openvpn 2.3.2 has memory leak? Hi, to bring this discussion to an end... On Thu, Nov 07, 2013 at 06:25:55PM +0800, Brad Zhang wrote: > Thank you so much. I am testing Openvpn 2.3.2 with your patch memory usage > some days. Here is result: Thanks for the exhaustive tests. This is really good to see, and it's impressive how much less memory PolarSSL need! The patch in question has been acked by David, has been pushed to git, and will be in 2.3.3 when we release that. [..] > I noticed your mail said if clients are disconnected, the memory will be > free, but when I tested openvpn + polarssl (no your patch), the memory did > not free completely, for example: > Openvpn 2.3.2 + Polarssl > 62672 55m 1408 S 0.0 2.8 0:06.91 openvpn (Have 200 connections) > 37864 34m 1412 S 0.0 1.7 0:06.84 openvpn (disconnect all connections) [..] > It is odd that memory do not free completely because I saw all the clients > disconnect correctly from Openvpn server logs. > I tried to disable the option "persist-key" and "persist-tun", it did not > make better when clients disconnected. Finding the reason for that is not trivial. What comes into play here is how the "malloc()" and "free()" functions works on that particular operating system - usually, not all malloc() requests fetch memory from the kernel directly, but usually malloc() will request a larger block (with sbrk() or mmap()), and maintain that locally in the process, saving on syscall overhead. free() will also work only locally first, and only if the library decides "it's worth returing something to the kernel" it will actually do so - so it's not unusual to never see memory usage actually go down if your library just keeps the memory chunks around... So it might be that in this case, there's a memory allocation right in the middle of a larger area which is still in use, so the larger area cannot be returned (but the memory might be re-used when the clients connect again, so is not "lost") - but without very extensive debugging, including fully understanding the way the local malloc()/free() implementations work, this is impossible to give a definite answer to. What we know, and this is good :-), is that openvpn *does* call free() on everything it allocates - this we can learn from valgrind, but it will only tell us how the situation is at program end, not how the memory layout looks like while it runs (I think it can be told to do so, but there are *many* short-lived malloc()/free() uses in OpenVPN, so looking at the log files would be a huge amount of work). Anyway. I think what we have is "good enough" given the amount of work we can invest here, and that there are so many other areas where we need to fix bugs... Thanks for bringing this up, and making us reconsider the buffer changes we did in best faith :-) gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de [attachment "att77ku7.dat" deleted by Arno Odermatt/R&D/SCH/SCHINDLER] ------------------------------------------------------------------------------ Shape the Mobile Experience: Free Subscription Software experts and developers: Be at the forefront of tech innovation. Intel(R) Software Adrenaline delivers strategic insight and game-changing conversations that shape the rapidly evolving mobile landscape. Sign up now. http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clktrk _________________________ ______________________ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel ----- Forwarded by Arno Odermatt/R&D/SCH/SCHINDLER on 22.11.2013 10:48 ----- From: arno.oderm...@ch.schindler.com To: Gert Doering <g...@greenie.muc.de>, Cc: openvpn-devel@lists.sourceforge.net List-Post: openvpn-devel@lists.sourceforge.net Date: 08.11.2013 11:56 Subject: Re: [Openvpn-devel] Does openvpn 2.3.2 has memory leak? Dear Gert, after having been able to spend a smooth operating time of the around 3400 permanent connected O-VPN clients (after having attached memory in form of a 10Gb swap partition), we just recognized a hickout, where all clients got disconnected. Unfortunately, after around 3270 disconnected clients, memory was'nt freed at all: 3270 /etc/openvpn/cgwvpn/persipoff top - 10:31:58 up 31 days, 20:10, 2 users, load average: 0.78, 0.70, 0.69 Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3%us, 23.2%sy, 0.0%ni, 50.0%id, 0.0%wa, 14.4%hi, 0.0%si, 12.0%st Mem: 7629920k total, 5073576k used, 2556344k free, 159504k buffers Swap: 10485756k total, 0k used, 10485756k free, 192536k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22810 root 20 0 4481m 4.2g 1996 S 62.5 58.1 5102:38 openvpn 12619 root 20 0 8 4 0 R 3.0 0.0 0:00.09 openvpn 12222 root 20 0 15228 1216 932 R 0.3 0.0 0:00.02 top Unfortunately, this is not something, we had expected. We have not applied the patch yet, which brings back the "2.1" behavior. freundlich grüsst, kind regards, _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Arno Odermatt Phone +41414454053 arno.oderm...@ch.schindler.com Schindler Elevator Ltd. | Corporate Research & Development | CRD/CO-BAT Zugerstr. 13 | CH-6030 Ebikon, Switzerland http://www.schindler.com _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Please consider your environment. Schindler supports sustainable urban development with safe, reliable and ecologically sound mobility solutions. From: Gert Doering <g...@greenie.muc.de> To: Brad Zhang <hebei5...@gmail.com>, Cc: openvpn-devel@lists.sourceforge.net List-Post: openvpn-devel@lists.sourceforge.net Date: 23.10.2013 16:20 Subject: Re: [Openvpn-devel] Does openvpn 2.3.2 has memory leak? Hi, On Sun, Oct 06, 2013 at 09:21:46PM +0800, Brad Zhang wrote: > Do you have some idea about this issue? Looking forward to your reply. Coming back to *this* thread, after some debugging in the other thread with Arno, I think what I can say so far is that we're observing two different things: - the first renegotiation brings a big jump in memory, but this is to be expected, because the new keying material needs extra memory, and the memory for the old key is not released right away (it seems to be released at the next renegotiation, or at client disconnect - I've seen VSZ/RSS go down(!) when clients disconnect, so that seems to work right). This is the "big" jump in memory consumption which is actually killing Arno's setup, because there is not enough virtual memory available to hold all that is needed for 3000 clients (but that can be fixed by increasing max-memory limits if there is enough RAM). - each further renegotiation leaks "a few kbyte" of memory per client, which should also be fully returned when the client disconnects - this is the issue I discovered earlier, with the GC handling of the per-client environment set, and I'll send a patch for that "soon" (as soon as I have fully understood the code involved). Now, what I do not really understand is why the memory consumption on the 3rd, 4th, 5th renegotiation seems to slow down - there seems to be some additional memory fragmentation involved in the 2nd+3rd renegotiation, which leads to memory re-use later on. Or so. Testing this with "reneg-sec 60" is actually quite interesting... here's *disconnect* of a client: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 19892 0.0 0.0 7052 4304 pts/7 SN+ 17:48 0:01 openvpn ../serv USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 19892 0.0 0.0 6300 3740 pts/7 SN+ 17:48 0:01 openvpn ../serv ... so you can see that memory is returned to the OS. (I think this will not be visible if many clients are connected as the free()ed memory will be "somewhere in the middle" - but disconnecting *all* clients should also show memory being returned) Now going to stare more at the code... :-) gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de [attachment "attxgxs6.dat" deleted by Arno Odermatt/R&D/SCH/SCHINDLER] ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk __________________ _________________________ ____ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel ****************************************************** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. ******************************************************* ------------------------------------------------------------------------------ November Webinars for C, C++, Fortran Developers Accelerate application performance with scalable programming models. Explore techniques for threading, error checking, porting, and tuning. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk _________________________ ______________________ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel ****************************************************** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. *******************************************************
