wired memory - again!
top reports wired memory 128MB WHERE it is used? below results of vmstat -m and vmstat -z values does not sum up even to half of it FreeBSD 9 - few days old. What i am missing and why there are SO MUCH wired memory on 1GB machine without X11 or virtualbox Type InUse MemUse HighUse Requests Size(s) hhook 2 1K -2 128 entropy 102464K - 1024 64 ithread7913K - 80 32,128,256 prison_racct 1 1K -1 512 acpiintr 1 1K -1 64 acpica 1475 144K -40951 16,32,64,128,256,512,1024 KTRACE 10013K - 100 128 pci_link16 2K - 16 32,128 linker 173 182K - 222 16,32,64,128,256,512,1024,2048,4096 lockf29 4K -43890 64,128,256 loginclass 2 1K - 798 64 temp3912K - 5630768 16,32,64,128,256,512,1024,2048,4096 devbuf 5877 8954K - 6191 16,32,64,128,256,512,1024,2048,4096 cache 1 1K -1 32 acpitask 1 2K -1 2048 module 19024K - 190 128 hdaa 536K -5 1024,2048 mtx_pool 216K -2 hdac 1 1K -1 1024 hdacc 1 1K -1 32 osd 2 1K -2 16,64 pmchooks 1 1K -1 128 subproc 975 715K - 3179367 512,4096 proc 2 2K -2 1024 session24 3K - 919 128 pgrp33 5K - 6485 128 cred 10216K - 16668512 64,256 uidinfo 8 2K - 823 128,256 plimit23 6K -30450 256 agp 1 1K -8 32,128 CAM periph 6 2K - 26 16,32,64,128,256 sysctltmp 0 0K - 129246 16,32,64,128 sysctloid 3473 171K - 3663 16,32,64,128 sysctl 0 0K - 412154 16,32,64 tidhash 1 2K -1 2048 callout 3 192K -3 umtx 2712 339K - 2784 128 p1003.1b 1 1K -1 16 SWAP 2 1097K -2 64 CAM XPT4725K - 320 16,32,64,128,256,1024,2048 bus-sc75 254K - 845 16,32,64,128,256,512,1024,2048 bus 56953K - 3061 16,32,64,128,256,1024 devstat 4 9K -4 32,4096 eventhandler73 6K - 73 64,128 kobj 118 472K - 184 4096 feeder20 2K - 95 32,128 Per-cpu 1 1K -1 32 ata_pci 2 1K -2 64 acpisem17 3K - 17 128 scsi_cd 0 0K -4 16 rman 18421K - 594 16,32,128 sbuf 1 1K - 565 16,32,64,128,256,512,1024,2048,4096 drm_drawable 0 0K -3 64 stack 0 0K -2 256 taskqueue17 2K - 17 16,32,128 Unitno29 2K - 2679811 32,64 iov 0 0K - 3495764 16,32,64,128,256,512,1024,2048,4096 select 1038 130K - 1039 128 ioctlops 0 0K - 4852031 16,32,64,128,256,512,1024,2048,4096 msg 4 398K -4 2048 sem 4 106K -4 2048,4096 shm 120K - 74 2048 tty 8 8K - 66 512,1024 pts 3 1K - 59 256 mbuf_tag 0 0K -18887 32,64 shmfd 1 8K -1 mixer 312K -3 4096 pcb2013K -10587 16,32,1024,2048,4096 soname15 2K -47317 16,32,64,128 acl 0 0K - 100159 4096 biobuf4284K -69042 2048 vfscache 1 512K -1 cl_savebuf 0 0K -46771 64,128,256,512,1024,2048 vfs_hash 1 256K -1 vnodes 3 1K - 66 64,256 drm_ctxbitmap 1 4K -1 4096 mount 197 9K - 484 16,32,64,128,256 vnodemarker 0 0K - 113570 512 BPF1410K - 14 128,512,4096 ether_multi 9 1K - 46 16,64 ifaddr3813K - 70 32,512,4096 ifnet1019K - 10 128,2048 drm_agplists 1 1K -1 128 clone 416K -4 4096 arpcom 2 1K -2 16 lltable11 5K - 46 256,512 drm_b
Re: ifconfig accepting hostname as ipv4 address
input. Moreover, ifconfig em0 some_valid_fqdn/MASK silently ignores it, so you can't set valid CIDR address using this notation. Classful era has ended more than 10 years ago, do we still want to keep this behavior? were not aware of that option, and it is rather stupid option - you should work on addresses not names when configuring network ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SuperPages utilization survey
On Fri, 1 Jun 2012 14:54:48 +0200 Ivan Voras wrote: IV> On 1 June 2012 14:35, Wojciech Puchar wrote: >>> http://people.freebsd.org/~ivoras/stuff/spsurvey.py ... IV> If anyone posts more data, I'll analyse it. I'm more worried about the IV> granularity of procstat, where it marks the entire region if a single IV> superpage exists in it - it means any such analysis is only IV> approximate. Here is a patch (for kernel and procstat) that allows to see amount of pages mapped to superpages. http://people.freebsd.org/~trociny/procstat-superpages.cnt.1.patch Not sure it is useful enough to be committed. -- Mikolaj Golub ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: ifconfig accepting hostname as ipv4 address
On Sat, Jun 9, 2012 at 12:23 AM, Wojciech Puchar wrote: >> input. >> Moreover, ifconfig em0 some_valid_fqdn/MASK silently ignores it, so you >> can't set valid CIDR address using this notation. >> >> Classful era has ended more than 10 years ago, do we still want to keep >> this behavior? >> > were not aware of that option, and it is rather stupid option - you should > work on addresses not names when configuring network I agree that it's not the best configuration in the world, as it would only work 100% if a machine had proper DNS records or a definitive hosts file. There are already enough bugs with static IP configurations and hostnames as-is *I'm looking at you mountlate* -- no sense to introduce more potentially buggy interoperability that only works in a handful of niche cases. Thanks, -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SuperPages utilization survey
On Sat, Jun 09, 2012 at 10:31:17AM +0300, Mikolaj Golub wrote: > > On Fri, 1 Jun 2012 14:54:48 +0200 Ivan Voras wrote: > > IV> On 1 June 2012 14:35, Wojciech Puchar > wrote: > >>> http://people.freebsd.org/~ivoras/stuff/spsurvey.py > > ... > > IV> If anyone posts more data, I'll analyse it. I'm more worried about the > IV> granularity of procstat, where it marks the entire region if a single > IV> superpage exists in it - it means any such analysis is only > IV> approximate. > > Here is a patch (for kernel and procstat) that allows to see amount of pages > mapped to superpages. > > http://people.freebsd.org/~trociny/procstat-superpages.cnt.1.patch > > Not sure it is useful enough to be committed. Superpage aggregates mappings for several normal-sized pages. As a consequence, when you iterate over small pages in sysctl_kern_proc_vmmap(), you account each superpage as many time as much constituent small pages it contains. pgpd9ymZ8zRqH.pgp Description: PGP signature
Re: SuperPages utilization survey
On Sat, 9 Jun 2012 11:38:22 +0300 Konstantin Belousov wrote: KB> On Sat, Jun 09, 2012 at 10:31:17AM +0300, Mikolaj Golub wrote: >> >> On Fri, 1 Jun 2012 14:54:48 +0200 Ivan Voras wrote: >> >> IV> On 1 June 2012 14:35, Wojciech Puchar >> wrote: >> >>> http://people.freebsd.org/~ivoras/stuff/spsurvey.py >> >> ... >> >> IV> If anyone posts more data, I'll analyse it. I'm more worried about the >> IV> granularity of procstat, where it marks the entire region if a single >> IV> superpage exists in it - it means any such analysis is only >> IV> approximate. >> >> Here is a patch (for kernel and procstat) that allows to see amount of pages >> mapped to superpages. >> >> http://people.freebsd.org/~trociny/procstat-superpages.cnt.1.patch >> >> Not sure it is useful enough to be committed. KB> Superpage aggregates mappings for several normal-sized pages. KB> As a consequence, when you iterate over small pages in KB> sysctl_kern_proc_vmmap(), you account each superpage as many time as KB> much constituent small pages it contains. This is exactly what my intention was to count: how much memory are handled by superpages (using normal-sized page as a measurement unit), not amount of superpages. And I think this is what Ivan wanted to know. Do you think it is better to return number of superpages? -- Mikolaj Golub ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SuperPages utilization survey
On Sat, Jun 09, 2012 at 12:03:43PM +0300, Mikolaj Golub wrote: > > On Sat, 9 Jun 2012 11:38:22 +0300 Konstantin Belousov wrote: > > KB> On Sat, Jun 09, 2012 at 10:31:17AM +0300, Mikolaj Golub wrote: > >> > >> On Fri, 1 Jun 2012 14:54:48 +0200 Ivan Voras wrote: > >> > >> IV> On 1 June 2012 14:35, Wojciech Puchar > wrote: > >> >>> http://people.freebsd.org/~ivoras/stuff/spsurvey.py > >> > >> ... > >> > >> IV> If anyone posts more data, I'll analyse it. I'm more worried about > the > >> IV> granularity of procstat, where it marks the entire region if a single > >> IV> superpage exists in it - it means any such analysis is only > >> IV> approximate. > >> > >> Here is a patch (for kernel and procstat) that allows to see amount of > pages > >> mapped to superpages. > >> > >> http://people.freebsd.org/~trociny/procstat-superpages.cnt.1.patch > >> > >> Not sure it is useful enough to be committed. > > KB> Superpage aggregates mappings for several normal-sized pages. > KB> As a consequence, when you iterate over small pages in > KB> sysctl_kern_proc_vmmap(), you account each superpage as many time as > KB> much constituent small pages it contains. > > This is exactly what my intention was to count: how much memory are handled by > superpages (using normal-sized page as a measurement unit), not amount of > superpages. And I think this is what Ivan wanted to know. Do you think it is > better to return number of superpages? > Well, if I see a report informing me that some 2M region contains 512 super pages, how should I interpret it ? For me, it is only one superpage (mapping) that can be created in one 2M region. pgpjIsQYdxpkt.pgp Description: PGP signature
Re: SuperPages utilization survey
On Sat, 9 Jun 2012 12:07:40 +0300 Konstantin Belousov wrote: KB> Well, if I see a report informing me that some 2M region contains 512 super KB> pages, how should I interpret it ? For me, it is only one superpage (mapping) KB> that can be created in one 2M region. Well, if I see a report like below: PID STARTEND PRTRES PRES SUP REF SHD FL TP PATH 485680x800c00x820c0 rw- 1310720 51712 2 0 --S df it tells me that for the region 0x800c0-0x820c0 (512Mb) we have 131072 * 4k = 512Mb resident and 51712 * 4k = 202Mb (a litle less than a half of the region) promoted (mapped) to superpages. If I had number of superpages here I would need additional knowledge (a superpage size) to calculate how effectively superpages are used. But actually, no much difference for me. To get a number of superpages is it enough just to divide the result obtained counting normal-sized pages by (2M/4k) factor? -- Mikolaj Golub ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SuperPages utilization survey
On 9 Jun 2012, at 10:46, Mikolaj Golub wrote: > On Sat, 9 Jun 2012 12:07:40 +0300 Konstantin Belousov wrote: > > KB> Well, if I see a report informing me that some 2M region contains 512 > super > KB> pages, how should I interpret it ? For me, it is only one superpage > (mapping) > KB> that can be created in one 2M region. > > Well, if I see a report like below: > > PID STARTEND PRTRES PRES SUP REF SHD FL > TP PATH > 485680x800c00x820c0 rw- 1310720 51712 2 0 --S > df > > it tells me that for the region 0x800c0-0x820c0 (512Mb) we have 131072 > * 4k = 512Mb resident and 51712 * 4k = 202Mb (a litle less than a half of the > region) promoted (mapped) to superpages. > > If I had number of superpages here I would need additional knowledge (a > superpage size) to calculate how effectively superpages are used. > > But actually, no much difference for me. To get a number of superpages is it > enough just to divide the result obtained counting normal-sized pages by > (2M/4k) factor? Remember also that superpage sizes are not necessarily 2M on all architectures, and in principle, many different page sizes might be simultaneously supported (e.g., on MIPS). I wonder if there's some way to capture that notion in the output somewhere so that, if we start supporting more granular page size control (something Alan might comment on), tool output doesn't need to be changed. Robert ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SuperPages utilization survey
On Sat, Jun 09, 2012 at 12:46:53PM +0300, Mikolaj Golub wrote: > > On Sat, 9 Jun 2012 12:07:40 +0300 Konstantin Belousov wrote: > > KB> Well, if I see a report informing me that some 2M region contains 512 > super > KB> pages, how should I interpret it ? For me, it is only one superpage > (mapping) > KB> that can be created in one 2M region. > > Well, if I see a report like below: > > PID STARTEND PRTRES PRES SUP REF SHD FL > TP PATH > 485680x800c00x820c0 rw- 1310720 51712 2 0 --S > df > > it tells me that for the region 0x800c0-0x820c0 (512Mb) we have 131072 > * 4k = 512Mb resident and 51712 * 4k = 202Mb (a litle less than a half of the > region) promoted (mapped) to superpages. > > If I had number of superpages here I would need additional knowledge (a > superpage size) to calculate how effectively superpages are used. > > But actually, no much difference for me. To get a number of superpages is it > enough just to divide the result obtained counting normal-sized pages by > (2M/4k) factor? First, there is nothing which would prevent demotion from happens while you iterate over the map, so you could get funyy numbers, like 42 superpages for 2M region with your method. Second, the superpage size if machine-depended, and even single machine could support differently sized superpage. For amd64, hardware can support 2M and 1G pages, and for i386 you can get 4M or 2M depending on PAE config. And last, I in fact do not see much use for any 'superpage count'. Would I would like to see is the TLB miss count for a region. Then I could estimate whether superpage enabling provided some advantage. Just as a note, if there were no accesses to a region after promotion, then promotion is the waste. Anyway, please do not consider this as discouraging you from doing a useful work. pgpv9Di6amoeA.pgp Description: PGP signature
Re: SuperPages utilization survey
On 9 Jun 2012, at 11:05, Konstantin Belousov wrote: > First, there is nothing which would prevent demotion from happens while > you iterate over the map, so you could get funyy numbers, like 42 superpages > for 2M region with your method. > > Second, the superpage size if machine-depended, and even single machine > could support differently sized superpage. For amd64, hardware can support > 2M and 1G pages, and for i386 you can get 4M or 2M depending on PAE config. > > And last, I in fact do not see much use for any 'superpage count'. Would I > would like to see is the TLB miss count for a region. Then I could estimate > whether superpage enabling provided some advantage. Just as a note, if > there were no accesses to a region after promotion, then promotion is > the waste. > > Anyway, please do not consider this as discouraging you from doing a > useful work. Despite the rendering and underlying semantic issues, I admit that I would like to know when superpages are being used by processes -- perhaps enough information to construct a histogram of page sizes for each mapping. Robert___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
cleaning /usr/obj before copying it to USB key
Hi, I build kernel and userland (out of SVN) and install in on an USB key mounted as /mnt this way: # cd /usr/src ... # make installworld DESTDIR=/mnt # make installkernel DESTDIR=/mnt KERNCONF=GENERIC INSTALL_NODEBUG=t # make distrib-dirs DESTDIR=/mnt # make distribution DESTDIR=/mnt To use the (booted) USB key later to install other laptops or netbooks I enrich the key with /usr/src and /usr/obj as: # cd /usr # cp -Rp src /mnt/usr # cp -Rp obj /mnt/usr Having done this I can use the key just to install the system on the laptop with the above DESTDIR=/mnt wherein /mnt is now the target root of the laptop; all this works just fine; my problem is that the both 'cp -Rp ...' commands takes many hours (12 and six hours) because they are transferring a lot(!!!) of small files; I have had a look into /usr/obj and it seems that after 'makeworld' and 'makekernel' there are left over a lot of temporary files from the build processes... is there a clean way to remove those files before 'cp -Rp obj /mnt/usr' while the result is still useful for another make install with DESTDIR=/mnt ? Thanks matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e - w http://www.unixarea.de/ UNIX since V7 on PDP-11 | UNIX on mainframe since ESER 1055 (IBM /370) UNIX on x86 since SVR4.2 UnixWare 2.1.2 | FreeBSD since 2.2.5 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: boot menu option to disable graphics mode
On Sat, Jun 09, 2012 at 07:28:50AM +0300, Andriy Gapon wrote: > on 09/06/2012 04:16 Jason Hellenthal said the following: > > runlevel support might be a better solution so it does not differ that > > much from what other systems do and would be easy for people to grasp. > > Patches are welcome, as always. > I agree... ;) How about generic runlevel support through kenv instead ? Set runlevel by default to 3 , where just like any other system is multiuser, and provide support in the rc scripts to look at kenv. While documenting "runlevel" in init(8)'s man page since that is where most people look for these things. This way a we could define a while bunch of things around generic runlevels and if perhaps runlevels ever make it into FreeBSD the support for them will already exist. -- - (2^(N-1)) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: boot menu option to disable graphics mode
On Thu, 07 Jun 2012 12:57:41 +0300, Gleb Kurtsou wrote: > On (07/06/2012 11:56), Andriy Gapon wrote: >> A user doesn't have to select the option unless he needs to. A "simple >> user" can just reboot without selecting the option to get back his X. A >> user doesn't have to learn anything about the code, just about kenv and >> "magic" inhibit_gui variable. > > What do you think about adding generic support for overriding *_enable > options in rc.conf? > > I'd like to be able to disable services at boot prompt, e.g. # set > rc.slim_enable="no" -- overrides slim_enable="yes" in rc.conf > > Similarly rc.pf_enable="no" > > Then introduce x_enable knob (=yes by default) to disable login > managers. User will be able to override this setting with # service xdm > forcestart > That's trivial to implement: http://lists.freebsd.org/pipermail/freebsd-current/2007-November/079241.html Still applies with minor reject that can be ignored or easily resolved. It also brings support for overriding path to rc.conf, allowing multiple boot configurations. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: cleaning /usr/obj before copying it to USB key
my problem is that the both 'cp -Rp ...' commands takes many hours (12 and six hours) because they are transferring a lot(!!!) of small files; I unless you want to kill you USB drive quickly (yes, it simulates disk THAT badly) then create image file on your hard disk exactly equal to your USB , use mdconfig, make all you need and then write with dd. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: boot menu option to disable graphics mode
On 06/07/2012 11:10, Andriy Gapon wrote: > on 07/06/2012 17:29 Doug Barton said the following: >> On 06/07/2012 02:57 AM, Gleb Kurtsou wrote: >>> What do you think about adding generic support for overriding *_enable >>> options in rc.conf? >>> >>> I'd like to be able to disable services at boot prompt, e.g. >>> # set rc.slim_enable="no" -- overrides slim_enable="yes" in rc.conf >>> >>> Similarly rc.pf_enable="no" >>> >>> Then introduce x_enable knob (=yes by default) to disable login >>> managers. User will be able to override this setting with >>> # service xdm forcestart >> >> Why not just: >> >> boot single user >> fsck -p >> mount -a >> $EDITOR /etc/rc.conf[.local] >> > > Ah, right. Why provide a way to do something using one command at one prompt > (or even toggling a menu option using a single keystroke) when you can already > do the same using multiple commands at multiple places (and also trying to not > forget to undo your changes later)... I realize you were being sarcastic, but your question deserves an answer. If this were a problem we didn't already have a solution for, I'd be much more interested in what you're proposing. But in no particular order ... 1. This is not something most users would have to do very often, if at all. 2. We have a variety of different login managers, several of which do things subtly differently, all of which would need ongoing support. 3. While the changes you're proposing sound simple, the startup stuff has some subtle interactions that we don't like to disrupt without good reason. It's also worth pointing out that if all you need is a shell at boot time, you can still do Ctrl-Alt-F1 to get that, without having to change anything. And if you find yourself needing to prevent the login manager from starting more often than not, just disable it by default and start it with 'service onestart', or use startx. My point being that this doesn't come with zero costs, and has very little benefit. That usually spells "no" in my book. Doug -- This .signature sanitized for your protection ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: cleaning /usr/obj before copying it to USB key
On Jun 9, 2012, at 7:35 AM, Matthias Apitz wrote:
>
> To use the (booted) USB key later to install other laptops or netbooks I
> enrich the key with /usr/src and /usr/obj as:
>
> # cd /usr
> # cp -Rp src /mnt/usr
> # cp -Rp obj /mnt/usr
>
> my problem is that the both 'cp -Rp ...' commands takes many hours (12
> and six hours) because they are transferring a lot(!!!) of small files;
As someone else pointed out, flash drives are pretty slow
when making a lot of small writes.
You can speed things up a lot by creating the image locally
then copying the image to USB. Here are parts of a shell
script I've been using for something similar:
# Create an empty file IMG_SIZE bytes
dd if=/dev/zero of=${IMG} bs=1 seek=${IMG_SIZE} count=0
# Attach it as a virtual disk device
MD=`mdconfig -a -t vnode -f ${IMG}`
# Partition the virtual disk
gpart create -s MBR ${MD}
gpart add -t freebsd ${MD}
gpart set -a active -i 1 ${MD}
# Format and mount
newfs ${MD}s1 >/dev/null
mount /dev/${MD}s1 /mnt
… copy stuff …
# Unmount and detach the virtual disk
umount /dev/${MD}s1
mdconfig -d -u ${MD}
# Copy the disk image to your flash drive
dd if=${IMG} of=${SDCARD} bs=8m
> I have had a look into /usr/obj and it seems that after 'makeworld' and
> 'makekernel' there are left over a lot of temporary files from the build
> processes...
>
> is there a clean way to remove those files before 'cp -Rp obj /mnt/usr'
> while the result is still useful for another make install with DESTDIR=/mnt ?
You can delete all of the '.o' files using a command like this:
find /usr/obj -name '*.o' | xargs rm
With a little experimenting, you should be able to extend this
to remove most of the intermediate files.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: wired memory - again!
On Sat, 2012-06-09 at 09:21 +0200, Wojciech Puchar wrote: > top reports wired memory 128MB > > > WHERE it is used? below results of vmstat -m and vmstat -z > values does not sum up even to half of it > FreeBSD 9 - few days old. > > What i am missing and why there are SO MUCH wired memory on 1GB machine > without X11 or virtualbox > > [vmstat output snipped] > I have been struggling to answer the same question for about a week on our embedded systems (running 8.2). We have systems with 64MB ram which have 20MB wired, and I couldn't find any way to directly view what that wired memory is being used for. I also discovered that the vmstat output accounted for only a tiny fraction of the 20MB. What I eventually determined is that there is some sort of correlation between vfs buffer space and wired memory. Our embedded systems typically do very little disk IO, but during some testing we were spewing debug output to /var/log/messages at the rate of several lines per second for hours. Under these conditions the amount of wired memory would climb from its usual of about 8MB to around 20MB, and once it climbed that high it pretty much never went down, or only went down a couple MB. The resulting memory pressure caused our apps to get killed over and over again with "out of swap space" (we have no swap on these systems). The kernel auto-tunes the vfs buffer space using the formula "for the first 64 MB of ram use 1/4 for buffers, plus 1/10 of the ram over 64 MB." Using 16 of 64 MB of ram for buffer space seems insane to me, but maybe it makes sense on certain types of servers or something. I added "option NBUF=128" to our kernel config and that dropped the buffer space to under 2 MB and since doing that I haven't seen the amount of wired memory ever go above 8 MB. I wonder whether my tuning of NBUF is affecting wired memory usage by indirectly tuning the 'nswbuf' value; I can't tune nswbuf directly because the embedded system is ARM-based and we have no loader(8) for setting tunablables. I'm not sure NBUF=128 is a good setting even for a system that doesn't do much IO, so I consider it experimental and we're testing under a variety of conditions to see if it leads to any unexpected behaviors. I'm certainly not suggesting anyone else rush to add this option to their kernel config. I am VERY curious about the nature of this correlation between vfs buffer space and wired memory. For the VM gurus: Is the behavior I'm seeing expected? Why would memory become wired and seemingly never get released back to one of the page queues after the IO is done? -- Ian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: wired memory - again!
On Sat, Jun 09, 2012 at 10:27:03AM -0600, Ian Lepore wrote: > On Sat, 2012-06-09 at 09:21 +0200, Wojciech Puchar wrote: > > top reports wired memory 128MB > > > > > > WHERE it is used? below results of vmstat -m and vmstat -z > > values does not sum up even to half of it > > FreeBSD 9 - few days old. > > > > What i am missing and why there are SO MUCH wired memory on 1GB machine > > without X11 or virtualbox > > > > [vmstat output snipped] > > > > > I have been struggling to answer the same question for about a week on > our embedded systems (running 8.2). We have systems with 64MB ram which > have 20MB wired, and I couldn't find any way to directly view what that > wired memory is being used for. I also discovered that the vmstat > output accounted for only a tiny fraction of the 20MB. > > What I eventually determined is that there is some sort of correlation > between vfs buffer space and wired memory. Our embedded systems > typically do very little disk IO, but during some testing we were > spewing debug output to /var/log/messages at the rate of several lines > per second for hours. Under these conditions the amount of wired memory > would climb from its usual of about 8MB to around 20MB, and once it > climbed that high it pretty much never went down, or only went down a > couple MB. The resulting memory pressure caused our apps to get killed > over and over again with "out of swap space" (we have no swap on these > systems). > > The kernel auto-tunes the vfs buffer space using the formula "for the > first 64 MB of ram use 1/4 for buffers, plus 1/10 of the ram over 64 > MB." Using 16 of 64 MB of ram for buffer space seems insane to me, but > maybe it makes sense on certain types of servers or something. I added > "option NBUF=128" to our kernel config and that dropped the buffer space > to under 2 MB and since doing that I haven't seen the amount of wired > memory ever go above 8 MB. I wonder whether my tuning of NBUF is > affecting wired memory usage by indirectly tuning the 'nswbuf' value; I > can't tune nswbuf directly because the embedded system is ARM-based and > we have no loader(8) for setting tunablables. > > I'm not sure NBUF=128 is a good setting even for a system that doesn't > do much IO, so I consider it experimental and we're testing under a > variety of conditions to see if it leads to any unexpected behaviors. > I'm certainly not suggesting anyone else rush to add this option to > their kernel config. > > I am VERY curious about the nature of this correlation between vfs > buffer space and wired memory. For the VM gurus: Is the behavior I'm > seeing expected? Why would memory become wired and seemingly never get > released back to one of the page queues after the IO is done? Hopefully, I can give you some information while you are waiting for answer from gurus. First, all memory allocated by UMA and consequently malloc(9) is wired. In other words, almost all memory used by kernel is accounted as wired. Second, the buffer cache wires the pages which are inserted into VMIO buffers. So your observation is basically right, cached buffers means that corresponding memory is removed from queues and put into wired state. When buffers are dissolved, pages are unwired and deactivated. This behaviour is in fact required by VFS, since you do expect to access buffer data when you get the buffer. pgpJWLzaFJDLZ.pgp Description: PGP signature
Re: nss compat shims
On 8-6-2012 22:06, Michael Bushkov wrote: > I don't know for sure, but it seems that there was no need to support > anything besides groups and password when nss_compat.c was committed. > At that time, IIRC, the modules that we had in ports supported only > these databases. Right. In the meantime hosts has been implemented in nss-pam-ldapd but currently does not work (at least on 9). It's been the author's concern too whether services/protocols etc is a needed feature, but I guess you won't find that out until you provide the feature. At least it seems logical to me to keep the nss_compat interface synchronized with the APIs that call nsdispatch. I guess I'll make patches for both. -- Mel ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: cleaning /usr/obj before copying it to USB key
El día Saturday, June 09, 2012 a las 08:57:33AM -0700, Tim Kientzle escribió: > On Jun 9, 2012, at 7:35 AM, Matthias Apitz wrote: > > > > To use the (booted) USB key later to install other laptops or netbooks I > > enrich the key with /usr/src and /usr/obj as: > > > > # cd /usr > > # cp -Rp src /mnt/usr > > # cp -Rp obj /mnt/usr > > > > my problem is that the both 'cp -Rp ...' commands takes many hours (12 > > and six hours) because they are transferring a lot(!!!) of small files; > > As someone else pointed out, flash drives are pretty slow > when making a lot of small writes. > > You can speed things up a lot by creating the image locally > then copying the image to USB. Here are parts of a shell > script I've been using for something similar: > ... Thanks for your hints, Wojciech and Tim; the USB key in question is of 16 marketing GByte; will this work as well with mdconfig or is there some limit? Thanks again matthias -- Matthias Apitz e - w http://www.unixarea.de/ UNIX since V7 on PDP-11, UNIX on mainframe since ESER 1055 (IBM /370) UNIX on x86 since SVR4.2 UnixWare 2.1.2, FreeBSD since 2.2.5 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: boot menu option to disable graphics mode
On 06/09/12 10:37, Jason Hellenthal wrote: On Sat, Jun 09, 2012 at 07:28:50AM +0300, Andriy Gapon wrote: on 09/06/2012 04:16 Jason Hellenthal said the following: runlevel support might be a better solution so it does not differ that much from what other systems do and would be easy for people to grasp. Patches are welcome, as always. I agree... ;) How about generic runlevel support through kenv instead ? I've wondered whether it would be more "BSD-sh" to specify a way to tell init, "Tell /etc/rc to run the scripts listed by rcorder up until we get NETWORKING." (Or SERVERS or whatever dependency you need, or "Stop just before LOGIN".) -- George Mitchell Set runlevel by default to 3 , where just like any other system is multiuser, and provide support in the rc scripts to look at kenv. While documenting "runlevel" in init(8)'s man page since that is where most people look for these things. This way a we could define a while bunch of things around generic runlevels and if perhaps runlevels ever make it into FreeBSD the support for them will already exist. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: wired memory - again!
The kernel auto-tunes the vfs buffer space using the formula "for the first 64 MB of ram use 1/4 for buffers, plus 1/10 of the ram over 64 i always limit it by kern.maxbcache=2000 on 1GB RAM machine with lots of I/O but actually with default MAXBSIZE and little I/O you can use safely 1MB It DOES NOT limit how much files are cached. But still - in spite of that i have a lot of wired memory. still it is bad nobody answered how to pin down "wired" memory use. instead of experimenting i would like rather to understand what is what. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: wired memory - again!
First, all memory allocated by UMA and consequently malloc(9) is wired. In other words, almost all memory used by kernel is accounted as wired. yes i understand this. still i found no way how to find out what allocated that much. Second, the buffer cache wires the pages which are inserted into VMIO buffers. So your observation is basically right, cached buffers means what are exactly "VMIO" buffers. i understand that page must be wired WHEN doing I/O. But i have too much wired memory even when doing no I/O at all. that corresponding memory is removed from queues and put into wired state. When buffers are dissolved, pages are unwired and deactivated. This behaviour is in fact required by VFS, since you do expect to access buffer data when you get the buffer. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: ifconfig accepting hostname as ipv4 address
On Sat, Jun 9, 2012 at 12:37 AM, Garrett Cooper wrote: > On Sat, Jun 9, 2012 at 12:23 AM, Wojciech Puchar > wrote: >>> input. >>> Moreover, ifconfig em0 some_valid_fqdn/MASK silently ignores it, so you >>> can't set valid CIDR address using this notation. >>> >>> Classful era has ended more than 10 years ago, do we still want to keep >>> this behavior? >>> >> were not aware of that option, and it is rather stupid option - you should >> work on addresses not names when configuring network > > I agree that it's not the best configuration in the world, as it > would only work 100% if a machine had proper DNS records or a > definitive hosts file. > There are already enough bugs with static IP configurations and > hostnames as-is *I'm looking at you mountlate* -- no sense to > introduce more potentially buggy interoperability that only works in a > handful of niche cases. The idea was that you could enter all of the local interface names in /etc/hosts and than just put the names into the ifconfig commands. It was handy for keeping track of what port connected where on systems that had numerous interfaces, though this was more common in the day of async serial lines and modems. I'll admit that I have mixed feelings about its practicality today, though it does not hurt anything, as far as I can tell. -- R. Kevin Oberman, Network Engineer E-mail: kob6...@gmail.com ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
VirtualBox on FreeBSD is looking for you!
Hi VirtualBox users! We are again at the point where I am kindly asking if someone is interested to help with the VirtualBox on FreeBSD work. We started with an active team of around 3 people but since about one year I ended up being a lonely ranger. Maintaining such a beast/high profile port by a single person is not possible for a longer period so we should really try to form a team to improve the situation. Additionally I started working on redports.org which requires more and more time so I cannot dedicate all my work to virtualbox. The situation of the port right now is not bad but I always end up fire fighting and only concentrate on serious problems due to the limited time I have. So it happens that people send bugreports and patches for virtualbox and don't get a response for weeks if at all. A lot of bugs that we have since day one are still present and the list of items that we should seriously do is getting longer. Many things of them are userland and porting stuff but there are also a lot of things to do in the kernel modules. Andriy Gapon has rewritten the r0 memory allocation stuff which significantly improved the situation in that area but the networking kernel modules are still in a bad shape. There are various known bugs (performance problems, instabilities, ...) in that area so it would be great if someone with networking expertise could have a look at the code. The USB stuff needs some love too. It is there but only works for a few special combinations of Device and Guest OS. Since VirtualBox 4.1 there is experimental support for PCI Passthrough so if we want that we need some Kernel API for the Intel/AMD IOMMU. I think the IOMMU code has already been written for BeHyVe so we need someone who puts all the stuff together and wants to find out how to integrate that in the kernel and vbox. The vbox developers offered their help on that but they need a Kernel API before we can start talking about it. So what is it that the virtualbox team needs to do? - regulary test latest SVN sources to find new problems early (build, runtime testing, create build fixes) - maintain all 8 ports (changes in CURRENT or other port updates keep breaking virtualbox around once per month) - update ports to new bugfix releases - review patches from the community and send them upstream then nag vbox developers to get them committed - help users to diagnose problems (help debugging, get stacktraces, collect information, give hints) - further porting efforts (coordinate and probably do it yourself) - optionsng adaptions - FreeBSD installer for the vbox additions to be able to build a VBoxAdditions.iso with FreeBSD support - implement vboxsf support (Shared Folders) - PCI Passthrough support - USB support (needs fixing) - Networking support (needs fixing) It is unrealistic that one single person can do a majority of these things so we seriously need a few people from different areas to improve the situation. Be it kernel developers, ports people or just power users that can help testing and diagnosing bugs. If you have an interest in VirtualBox on FreeBSD and a few spare cycles please get in contact with us (me?) to coordinate the further steps. I will do my best to help answering questions and help you with your first steps in vbox land. Don't worry if you think you are not experienced enough for the task. We all started that way and you get the chance to learn a lot - it just needs some time to get used to it. I have also created a dedicated VirtualBox on FreeBSD channel on freenode: #freebsd-vbox so you're welcome to join us! -- Bernhard Froehlich http://www.bluelife.at/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
