On Fri, Jun 03, 2011 at 12:39:40AM +0200, Torfinn Ingolfsen wrote: > On Thu, 02 Jun 2011 12:50:26 -0700 > Jeremy Chadwick <free...@jdc.parodius.com> wrote: > > > > This is a well-known thing with ZFS on FreeBSD. Because you're running > > 8.1-STABLE, this makes figuring out all the tunables and so on a lot > > more difficult than if you were running 8.2-STABLE. > > FWIW, the machine has been quite stable for me for a long time.
I've tried to explain this to people in the past (not sure if I did to you or not), but nobody ever listens. :-) You can't tune ZFS on FreeBSD, let it run for 24 hours -- or even 2 months -- and say "it's stable!" The exact same problem (specifically the kmem exhaustion issue) can appear many months later. It all depends on load, I/O history, and many other conditionals. Case in point. There are certain conditions where ZFS will take up more memory than just what's in the ARC. E.g. if you limit the ARC to 2GB and use top, on occasion you'll see ZFS's ARC usage (usually "Wired", but that also contains non-ZFS stuff; good luck segregating it!) at maybe 2.5GB. This is Normal(tm) and Expected(tm). So when tuning the ARC, you need to take into consideration that situation. This is why I tell people to avoid setting their ARC size to something "too close" to vm.kmem_size. You really do have to give ZFS "some room" memory-wise. This advice is with 8.2-STABLE in mind. The same advice applies to 8.1-STABLE, but different tunings are required. > > Please provide: > > > > 1) Contents of /boot/loader.conf > > root@kg-f2# more /boot/loader.conf > zfs_load="YES" > vfs.root.mountfrom="zfs:zroot" > siis_load="YES" > amdtemp_load="YES" > # testing without MSI > hw.pci.enable_msix="0" > hw.pci.enable_msi="0" Now I'm talking about 8.1-STABLE, but I'm going purely off of memory here, and like I said in a previous Email at this point in time I've focused on remembering tunings for 8.2-STABLE, so please keep that in mind. The above confirms you didn't tune vm.kmem_size and vm.kmem_size_max, nor did you try to limit the ARC (vfs.zfs.arc_max). Therefore I'm not surprised you're seeing said panic. Keep reading for tuning advice. > > 2) Output from: sysctl hw.physmem hw.usermem hw.realmem (your hardware > > page says 4GB, but I can't be bothered to sift through multi-pages > > of wiki documents and links to find the answers) > > root@kg-f2# sysctl hw.physmem hw.usermem hw.realmem > hw.physmem: 4141920256 > hw.usermem: 3721527296 > hw.realmem: 4966055936 Thanks -- this helps. > > 3) Output from: sysctl vfs.zfs.zio.use_uma > > root@kg-f2# sysctl vfs.zfs.zio.use_uma > vfs.zfs.zio.use_uma: 0 Cool, it's already off on your system, which is good. > > The "trick" is to tune /boot/loader.conf until you can gain stability. > > Well, the server has been reasonably stable for me for about a year now > (I had to replace a failing hard drive, but I count that as "wear" not > "instability"). I'm amazed it's run for a year. The I/O on this system must be very light, maybe kernel memory pressure hasn't been very high (until now; e.g. maybe other applications on the machine haven't grown in size) or you must have been **extremely** lucky. It's impossible to explain why it took a year, but it's pretty obvious why the problem happened. > > Again, because you're running 8.1-STABLE, the tuning parameters here > > will behave different than on 8.2-STABLE. We can go over those in a > > follow-up thread. > > I have no trouble with upgrading the server to 8.2-stable, if now is a good > time to do it. > (I haven't watched closely for any zfs related problems on the mailing list > lately.) > > > I've gotten to the point where I literally cannot remember all of the > > different situations/conditions/tunings for each FreeBSD kernel build, > > release, date, type, etc., so I tend to focus on the most recent > > RELENG_8 build. Then someone comes along with an older build..... > > Hehe. :-) > > I know what you mean. Keeping up with all this "stuff" is getting harder > every year. :) So to circle back around to the tuning advice: All of this below applies to amd64, by the way. Yes it matters. On 8.1-STABLE, you're going to need to tune the following settings in /boot/loader.conf to try and gain stability. Please note that the ZFS ARC limiting code in 8.1-STABLE is extremely (EXTREMELY!) different than in 8.2-STABLE; many bugs with the ARC limiting were fixed. What I'm saying here is that the tunings I'm about to give you will work, but you may still see the panic happen in certain situations. You can minimise this chance by going to 8.2-STABLE. Anyway, your system has 4GB of RAM installed in it, so on 8.1-STABLE I'd recommend you try these settings: vm.kmem_size="3584M" vm.kmem_size_max="3584M" vfs.zfs.arc_max="2048M" Now, these are all chosen by me off the top of my head with absolutely ZERO knowledge of what the memory usage on this system is like **without** ZFS in the picture. I'm making a lot of assumptions, and I'm assuming worst-case scenarios. For example, if this machine also runs mysqld and its tuned to take up a lot of memory, I would advocate dropping vfs.zfs.arc_max to 1536M or 1024M. Please don't drop it "too much"; ZFS performs best when it has lots of ARC. Since you mentioned going to 8.2-STABLE, all you need to tune on that version is one single tunable: vfs.zfs.arc_max Yup, that simple. The kmem "stuff" was greatly improved (big thanks to Alan Cox for that!), and some other adjustments were made (such as the changing the default value for vm.kmem_size_scale). HTH. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
