Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
On Wed, 2012-10-31 at 13:38 -0700, Adrian Chadd wrote: > On 31 October 2012 12:06, Konstantin Belousov wrote: > > > Watchdogd was recently changed to mlock its memory. This is the cause > > of the RSS increase. > > > > If not wired, swapout might cause a delay of the next pat, leading to > > panic. > > Right, but look at the virtual size of the 6.4 process. It's not 10 > megabytes at all. Even if you wired all of that into memory, it > wouldn't be 10 megabytes. > > > > Adrian After gathering some more evidence, I agree that the huge increase I noticed in watchdogd is caused by a combo of jemalloc's behavior and the recent addition of mlockall(2) to watchdogd. Since this is only slightly tangentially related to the OP's questions as near as I can tell, I've entered a PR for it[1], and we can followup with a separate discusssion thread about that. While jemalloc can explain the growth in VSZ between 6.4 and 9.x, it doesn't look like mlockall() has anything to do with the original question of why the RSZ got so much bigger. In other words, part of the original question is still unanswered. [1] http://www.freebsd.org/cgi/query-pr.cgi?pr=173332 -- Ian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
On Wed, Oct 31, 2012 at 2:06 PM, Konstantin Belousov wrote: > On Wed, Oct 31, 2012 at 11:52:06AM -0700, Adrian Chadd wrote: > > On 31 October 2012 11:20, Ian Lepore > wrote: > > > I think there are some things we should be investigating about the > > > growth of memory usage. I just noticed this: > > > > > > Freebsd 6.2 on an arm processor: > > > > > > 369 root 1 8 -88 1752K 748K nanslp 3:00 0.00% watchdogd > > > > > > Freebsd 10.0 on the same system: > > > > > > 367 root 1 -52 r0 10232K 10160K nanslp 10:04 0.00% watchdogd > > > > > > The 10.0 system is built with MALLOC_PRODUCTION (without that defined > > > the system won't even boot, it only has 64MB of ram). That's a crazy > > > amount of growth for a relatively simple daemon. > > > > Would you please, _please_ do some digging into this? > > > > It's quite possible there's something in the libraries that are > > allocating some memory upon first call invocation - yes, that's > > jemalloc, but it could also be other things like stdio. > > > > We really, really need to fix this userland bloat; it's terribly > > ridiculous at this point. There's no reason a watchdog daemon should > > take 10megabytes of RAM. > Watchdogd was recently changed to mlock its memory. This is the cause > of the RSS increase. > > Is it also statically linked? Alan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
On Sat, Nov 03, 2012 at 01:11:17PM -0500, Alan Cox wrote: > On Wed, Oct 31, 2012 at 2:06 PM, Konstantin Belousov > wrote: > > > On Wed, Oct 31, 2012 at 11:52:06AM -0700, Adrian Chadd wrote: > > > On 31 October 2012 11:20, Ian Lepore > > wrote: > > > > I think there are some things we should be investigating about the > > > > growth of memory usage. I just noticed this: > > > > > > > > Freebsd 6.2 on an arm processor: > > > > > > > > 369 root 1 8 -88 1752K 748K nanslp 3:00 0.00% watchdogd > > > > > > > > Freebsd 10.0 on the same system: > > > > > > > > 367 root 1 -52 r0 10232K 10160K nanslp 10:04 0.00% watchdogd > > > > > > > > The 10.0 system is built with MALLOC_PRODUCTION (without that defined > > > > the system won't even boot, it only has 64MB of ram). That's a crazy > > > > amount of growth for a relatively simple daemon. > > > > > > Would you please, _please_ do some digging into this? > > > > > > It's quite possible there's something in the libraries that are > > > allocating some memory upon first call invocation - yes, that's > > > jemalloc, but it could also be other things like stdio. > > > > > > We really, really need to fix this userland bloat; it's terribly > > > ridiculous at this point. There's no reason a watchdog daemon should > > > take 10megabytes of RAM. > > Watchdogd was recently changed to mlock its memory. This is the cause > > of the RSS increase. > > > > > Is it also statically linked? No. I do not think that it is reasonable to statically link watchdogd. It might result in some memory saving, but I dislike the whole idea of static linkage on Tier 1 platforms. pgpNDz1ZE5tte.pgp Description: PGP signature
watchdogd, jemalloc, and mlockall
In an attempt to un-hijack the thread about memory usage increase between 6.4 and 9.x, I'm starting a new thread here related to my recent discovery that watchdogd uses a lot more memory since it began using mlockall(2). I tried statically linking watchdogd and it made a small difference in RSS, presumably because it doesn't wire down all of libc and libm. VSZ RSS 10236 10164 Dynamic 8624 8636 Static Those numbers are from ps -u on an arm platform. I just updated the PR (bin/173332) with some procstat -v output comparing with/without mlockall(). It appears that the bulk of the new RSS bloat comes from jemalloc allocating vmspace in 8MB chunks. With mlockall(MCL_FUTURE) in effect that leads to wiring 8MB to satisfy what probably amounts to a few hundred bytes of malloc'd memory. It would probably also be a good idea to remove the floating point from watchdogd to avoid wiring all of libm. The floating point is used just to turn the timeout-in-seconds into a power-of-two-nanoseconds value. There's probably a reasonably efficient way to do that without calling log(), considering that it only happens once at program startup. -- Ian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: watchdogd, jemalloc, and mlockall
On Sat, Nov 03, 2012 at 12:38:39PM -0600, Ian Lepore wrote: > In an attempt to un-hijack the thread about memory usage increase > between 6.4 and 9.x, I'm starting a new thread here related to my recent > discovery that watchdogd uses a lot more memory since it began using > mlockall(2). > > I tried statically linking watchdogd and it made a small difference in > RSS, presumably because it doesn't wire down all of libc and libm. > > VSZ RSS > 10236 10164 Dynamic > 8624 8636 Static > > Those numbers are from ps -u on an arm platform. I just updated the PR > (bin/173332) with some procstat -v output comparing with/without > mlockall(). > > It appears that the bulk of the new RSS bloat comes from jemalloc > allocating vmspace in 8MB chunks. With mlockall(MCL_FUTURE) in effect > that leads to wiring 8MB to satisfy what probably amounts to a few > hundred bytes of malloc'd memory. > > It would probably also be a good idea to remove the floating point from > watchdogd to avoid wiring all of libm. The floating point is used just > to turn the timeout-in-seconds into a power-of-two-nanoseconds value. > There's probably a reasonably efficient way to do that without calling > log(), considering that it only happens once at program startup. No, I propose to add a switch to turn on/off the mlockall() call. I have no opinion on the default value of the suggested switch. pgp7Jg9fdTcDk.pgp Description: PGP signature
Re: watchdogd, jemalloc, and mlockall
On Sat, 2012-11-03 at 20:41 +0200, Konstantin Belousov wrote: > On Sat, Nov 03, 2012 at 12:38:39PM -0600, Ian Lepore wrote: > > In an attempt to un-hijack the thread about memory usage increase > > between 6.4 and 9.x, I'm starting a new thread here related to my recent > > discovery that watchdogd uses a lot more memory since it began using > > mlockall(2). > > > > I tried statically linking watchdogd and it made a small difference in > > RSS, presumably because it doesn't wire down all of libc and libm. > > > > VSZ RSS > > 10236 10164 Dynamic > > 8624 8636 Static > > > > Those numbers are from ps -u on an arm platform. I just updated the PR > > (bin/173332) with some procstat -v output comparing with/without > > mlockall(). > > > > It appears that the bulk of the new RSS bloat comes from jemalloc > > allocating vmspace in 8MB chunks. With mlockall(MCL_FUTURE) in effect > > that leads to wiring 8MB to satisfy what probably amounts to a few > > hundred bytes of malloc'd memory. > > > > It would probably also be a good idea to remove the floating point from > > watchdogd to avoid wiring all of libm. The floating point is used just > > to turn the timeout-in-seconds into a power-of-two-nanoseconds value. > > There's probably a reasonably efficient way to do that without calling > > log(), considering that it only happens once at program startup. > > No, I propose to add a switch to turn on/off the mlockall() call. > I have no opinion on the default value of the suggested switch. In a patch I submitted along with the PR, I added code to query the vm.swap_enabled sysctl and only call mlockall() when swapping is enabled. Nobody yet has said anything about what seems to me to be the real problem here: jemalloc grabs 8MB at a time even if you only need to malloc a few bytes, and there appears to be no way to control that behavior. Or maybe there's a knob in there that didn't jump out at me on a quick glance through the header files. -- Ian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: watchdogd, jemalloc, and mlockall
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/3/12 11:38 AM, Ian Lepore wrote: > In an attempt to un-hijack the thread about memory usage increase > between 6.4 and 9.x, I'm starting a new thread here related to my > recent discovery that watchdogd uses a lot more memory since it > began using mlockall(2). > > I tried statically linking watchdogd and it made a small difference > in RSS, presumably because it doesn't wire down all of libc and > libm. Speaking for this, the last time I brought this up, someone (can't remember, I think it was phk@) argued that the shared library would use only one copy of memory, while statically linked ones would be duplicated and thus use more memory. I haven't yet tried to prove or challenge that, though. Cheers, -BEGIN PGP SIGNATURE- iQEcBAEBCAAGBQJQlXeTAAoJEG80Jeu8UPuz7AUIAJOn67ETS7uHuIaPNByr9R6l S6l8uhwqTOsF+4jmuuDmjI25uiCAN4a3OU8i4n/ZGuarlip2Rr4BFWf+FUkkzdyk qButTuWC/agpuKofJ/7UubTXIEhpViWY/J2mqQTwgk+zeQ0bl2yjaqaR4hH3/ivi DQ3FWGzBhWD0Ohx/B0f33i9wvc5mCTTR5oxM78xvrQIPejG3lQHcwgmsd5XLgAuW 54UEEnklxAYLDf9eCsDo9nSsXQBKidmZop3ELtg08gUxtu5Ncf1+QraLxjdFzdr7 RrmQgcR4QrVtQeezWCRx2Y8VzGl0rtOunmQguNgkwRLo3KQlIU4IhpnaNrNez74= =HAd6 -END PGP SIGNATURE- ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: watchdogd, jemalloc, and mlockall
On Sat, 2012-11-03 at 12:59 -0700, Xin Li wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On 11/3/12 11:38 AM, Ian Lepore wrote: > > In an attempt to un-hijack the thread about memory usage increase > > between 6.4 and 9.x, I'm starting a new thread here related to my > > recent discovery that watchdogd uses a lot more memory since it > > began using mlockall(2). > > > > I tried statically linking watchdogd and it made a small difference > > in RSS, presumably because it doesn't wire down all of libc and > > libm. > > Speaking for this, the last time I brought this up, someone (can't > remember, I think it was phk@) argued that the shared library would > use only one copy of memory, while statically linked ones would be > duplicated and thus use more memory. I haven't yet tried to prove or > challenge that, though. That sounds right to me... if 3 or 4 daemons were to eventually be statically linked because of mlockall(), then each of them would have its own private copy of strlen(), and malloc(), and so on; we'd be back to the bad old days before shared libs came along. Each program would contain its own copy of only the routines from the library that it uses, not the entire library in each program. On the other hand, if even one daemon linked with shared libc uses mlockall(), then all of libc gets wired. As I understand it, only one physical copy of libc would exist in memory, still shared by almost all running apps. The entire contents of the library would continuously occupy physical memory, even the parts that no apps are using. It's hard to know how to weigh the various tradeoffs. I suspect there's no one correct answer. -- Ian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Hm, can you disable mlockall just to see what effect it has? Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Subversion - Sync Branch with Trunk
Hello, During my GSoC project, I branched HEAD in order to use it for the development of the client side part of my project. After some changes, I tried to sync my branch with HEAD but I have faced an error. Now, I am trying once again to sync my branch with HEAD, but I get the exact same error. The error is the following: svn: E175002: PROPFIND of '/socsvn/!svn/bc/236241/mirror/FreeBSD/head/sys/dev/usb/controller': 207 Multi-Status (https://socsvn.freebsd.org) svn: E175002: Error reading spooled REPORT request response The error appears after 2 hours of syncing my branch with HEAD, with U (updated) as the svn status code for most of the files, and 4-6 of them that I resolved the conflicts by selecting tf (theirs-full). This is what I do to sync my branch with trunk (as described in the SVN Book [1]): # the root of my working copy, it contains the .svn, client-side, and server-side directories cd /home/tzabal/akcrs svn status if [ no local modifications reported ]; then svn merge https://socsvn.freebsd.org/socsvn/mirror/FreeBSD/head client-side/akcrs-head fi Can you propose any solutions in order to sync my branch with HEAD? Regards [1] Keeping a Branch in Sync, http://svnbook.red-bean.com/en/1.6/svn.branchmerge.basicmerging.html P.S. The project's code is located at http://svnweb.freebsd.org/socsvn/soc2012/tzabal/ -- Tzanetos Balitsaris This message was sent using IMP, the Internet Messaging Program. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
