Superpages on amd64 FreeBSD 7.2-STABLE
We have a squid proxy process with very large memory requirements (10 - 20 GB) on a machine with 24GB of RAM. Unfortunately, we have to rotate the logs of this process once per day. When we do, it fork()s and exec()s about 16-20 child processes as helpers. Since it's got this multi-million-entry page table, that's a disaster, because it has to copy all those page table entries for each child, then throw them out. This takes a couple of minutes of 100% CPU usage, during which time the machine is pretty much unresponsive. Someone on the squid list suggested we try the new superpages feature (vm.pmap.pg_ps_enabled) in 7.2. We did, and after some tuning, we got it to work. Here's some "sysctl vm.pmap" for a similar machine with 16GB of RAM that does NOT have this setting enabled: vm.pmap.pv_entry_count: 2307899 vm.pmap.pde.promotions: 0 vm.pmap.pde.p_failures: 0 vm.pmap.pde.mappings: 0 vm.pmap.pde.demotions: 0 vm.pmap.pv_entry_max: 4276871 vm.pmap.pg_ps_enabled: 0 Now, here is the machine that does have it, just prior to the daily rotation mentioned above: vm.pmap.pv_entry_count: 61361 vm.pmap.pde.promotions: 23123 vm.pmap.pde.p_failures: 327946 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 17848 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 So it obviously this feature makes a huge difference and is a brilliant idea. :-) My (limited) understanding is that one of the primary benefits of this feature is to help situations like ours... a page table that's 512x smaller can be copied 512x faster. However, in practice this doesn't happen. It's like fork() breaks up the squid process into 4kb pages again. Here's the same machine's entries just after rotation: vm.pmap.pv_entry_count: 1908056 vm.pmap.pde.promotions: 23212 vm.pmap.pde.p_failures: 413171 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 21470 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 So some 3,600 superpages spontaneously turned into 1,850,000 4k pages. Once this happens, squid seems reluctant to use more superpages until its restarted. We get a lot of p_failures and a slow-but-steady stream of demotions. Here's the same machine just now: vm.pmap.pv_entry_count: 2022786 vm.pmap.pde.promotions: 25281 vm.pmap.pde.p_failures: 996027 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 21683 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 And a few minutes later: vm.pmap.pv_entry_count: 2021556 vm.pmap.pde.promotions: 25331 vm.pmap.pde.p_failures: 1001773 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 21684 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 (There were *no* p_failures or demotions in the several hours prior to rotation.) This trend continues... the pv_entry_count bounces up and down even though memory usage is increasing, so it's like it's trying to recover and convert things back (promotions), but it's having a lot of trouble (p_failures). It's not clear to me if this might be a problem with the superpages implementation, or if squid does something particularly horrible to its memory when it forks to cause this, but I wanted to ask about it on the list in case somebody who understands it better might know whats going on. :-) This is on FreeBSD-STABLE 7.2 amd64 r198976M. Thanks! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Wine on amd64 in 32 bit jail
On Mon, Nov 23, 2009 at 12:11:02AM -0500, Dylan Cochran wrote: > On Sat, Nov 21, 2009 at 5:52 AM, Peter Jeremy wrote: > > I did run into problems initially because my i386 userland wasn't > > aligned with my amd64 kernel but rebuilding both fixed that (I'm > > running 8.0-RC1 and a bit). > > > > Note that some tools that poke around in kernel innards won't work - > > ps and lsof are the most obvious. ktrace works but the resultant > > ktrace.out files need to read with an amd64 kdump. > > A word of caution: not all the ioctl's and setsockopt's have the > proper glue code under COMBAT_FREEBSD32... That was actually a nice typo :P G'luck, Peter -- Peter Pentchev r...@ringlet.netr...@space.bgr...@freebsd.org PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 This sentence would be seven words long if it were six words shorter. pgptwi6U2AKJJ.pgp Description: PGP signature
Re: Superpages on amd64 FreeBSD 7.2-STABLE
Is squid multithreaded? My first guess would be that you have one thread forking off all of these processes while other threads are still doing work and writing to different parts of the address space. I don't know the details of the superpages implementation but I could definitely see that this could be a problem. When the process is forked off the vm layer has to mark all of the pages in the parent process as copy-on-write. If there are other threads in squid writing to memory in this time, the vm layer will have to allocate a new page of memory for every page that squid writes to. This breaks up the superpage(superpages must be physically contiguous in memory). I don't know if the superpage implementation tries to alleviate this situation at all. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Thu, Nov 26, 2009 at 10:34 AM, Ryan Stone wrote: > Is squid multithreaded? No, it isn't: PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 75086 squid 1 40 12571M 12584M kqread 6 31:31 0.68% squid Thanks! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
Linda Messerschmidt writes: > Unfortunately, we have to rotate the logs of this process once per > day. When we do, it fork()s and exec()s about 16-20 child processes > as helpers. s/fork/vfork/ and you should be fine. DES -- Dag-Erling Smørgrav - d...@des.no ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
Dag-Erling Smørgrav writes: > Linda Messerschmidt writes: > > Unfortunately, we have to rotate the logs of this process once per > > day. When we do, it fork()s and exec()s about 16-20 child processes > > as helpers. > s/fork/vfork/ and you should be fine. ...and you should look into replacing Squid with Varnish, of course. DES -- Dag-Erling Smørgrav - d...@des.no ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
Hi Linda, vfork() should mitigate this -- i suggest replacing. respectfully, =jt On Thu, Nov 26, 2009 at 10:47, Linda Messerschmidt wrote: > On Thu, Nov 26, 2009 at 10:34 AM, Ryan Stone wrote: >> Is squid multithreaded? > > No, it isn't: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 75086 squid 1 4 0 12571M 12584M kqread 6 31:31 0.68% squid > > Thanks! > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
I think I was not clear with my message, I apologize. I did not mean to suggest that we were asking for help solving a problem with squid rotation. I provided that information as background to discuss what we observed as a potential misbehavior in the new VM superpages feature, in the hope that if there is a problem with the new feature, we can help find/resolve it or, if this is working as intended, hopefully gain some insight as to what's going on. Thanks! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
2009/11/26 Linda Messerschmidt > We have a squid proxy process with very large memory requirements (10 > - 20 GB) on a machine with 24GB of RAM. > > Unfortunately, we have to rotate the logs of this process once per > day. When we do, it fork()s and exec()s about 16-20 child processes > as helpers. Since it's got this multi-million-entry page table, > that's a disaster, because it has to copy all those page table entries > for each child, then throw them out. This takes a couple of minutes > of 100% CPU usage, during which time the machine is pretty much > unresponsive. > > Someone on the squid list suggested we try the new superpages feature > (vm.pmap.pg_ps_enabled) in 7.2. We did, and after some tuning, we got > it to work. > > Here's some "sysctl vm.pmap" for a similar machine with 16GB of RAM > that does NOT have this setting enabled: > > vm.pmap.pv_entry_count: 2307899 > vm.pmap.pde.promotions: 0 > vm.pmap.pde.p_failures: 0 > vm.pmap.pde.mappings: 0 > vm.pmap.pde.demotions: 0 > vm.pmap.pv_entry_max: 4276871 > vm.pmap.pg_ps_enabled: 0 > > Now, here is the machine that does have it, just prior to the daily > rotation mentioned above: > > vm.pmap.pv_entry_count: 61361 > vm.pmap.pde.promotions: 23123 > vm.pmap.pde.p_failures: 327946 > vm.pmap.pde.mappings: 1641 > vm.pmap.pde.demotions: 17848 > vm.pmap.pv_entry_max: 7330186 > vm.pmap.pg_ps_enabled: 1 > > So it obviously this feature makes a huge difference and is a > brilliant idea. :-) > > My (limited) understanding is that one of the primary benefits of this > feature is to help situations like ours... a page table that's 512x > smaller can be copied 512x faster. However, in practice this doesn't > happen. It's like fork() breaks up the squid process into 4kb pages > again. Here's the same machine's entries just after rotation: > > vm.pmap.pv_entry_count: 1908056 > vm.pmap.pde.promotions: 23212 > vm.pmap.pde.p_failures: 413171 > vm.pmap.pde.mappings: 1641 > vm.pmap.pde.demotions: 21470 > vm.pmap.pv_entry_max: 7330186 > vm.pmap.pg_ps_enabled: 1 > > So some 3,600 superpages spontaneously turned into 1,850,000 4k pages. > > Once this happens, squid seems reluctant to use more superpages until > its restarted. We get a lot of p_failures and a slow-but-steady > stream of demotions. Here's the same machine just now: > > vm.pmap.pv_entry_count: 2022786 > vm.pmap.pde.promotions: 25281 > vm.pmap.pde.p_failures: 996027 > vm.pmap.pde.mappings: 1641 > vm.pmap.pde.demotions: 21683 > vm.pmap.pv_entry_max: 7330186 > vm.pmap.pg_ps_enabled: 1 > > And a few minutes later: > > vm.pmap.pv_entry_count: 2021556 > vm.pmap.pde.promotions: 25331 > vm.pmap.pde.p_failures: 1001773 > vm.pmap.pde.mappings: 1641 > vm.pmap.pde.demotions: 21684 > vm.pmap.pv_entry_max: 7330186 > vm.pmap.pg_ps_enabled: 1 > > (There were *no* p_failures or demotions in the several hours prior to > rotation.) > > This trend continues... the pv_entry_count bounces up and down even > though memory usage is increasing, so it's like it's trying to recover > and convert things back (promotions), but it's having a lot of trouble > (p_failures). > > It's not clear to me if this might be a problem with the superpages > implementation, or if squid does something particularly horrible to > its memory when it forks to cause this, but I wanted to ask about it > on the list in case somebody who understands it better might know > whats going on. :-) > > This is on FreeBSD-STABLE 7.2 amd64 r198976M. > > Thanks! > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > Im sure you will get a lot of lovely answers to this but best keep things simple. WHy not just syslog it of to another server and offload all the compression to that box. You could even back it with zfs nad do on the fly gzip compression at the file system level, or use syslog-ng to do it. If you are worried about zfs and bsd use (open)*solaris or another filesystem with with inline compression ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Superpages on amd64 FreeBSD 7.2-STABLE
On Fri, 27 Nov 2009, krad wrote: > Im sure you will get a lot of lovely answers to this but best keep > things simple. WHy not just syslog it of to another server and > offload all the compression to that box. You could even back it with > zfs nad do on the fly gzip compression at the file system level, or > use syslog-ng to do it. If you are worried about zfs and bsd use > (open)*solaris or another filesystem with with inline compression Or send squids logs to a small buffer process which you then HUP when rotating logs. Also, I don't really understand why squid would fork when you tell it to rotate, seems like a design defect. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C signature.asc Description: This is a digitally signed message part.
