Re: ZFS log device removal?
On 19 Oct, 2010, at 07:24 , Adam Stylinski wrote: > I hope I'm not asking anything too obvious, but I just saw this commit via > the svn-src-all list: > http://lists.freebsd.org/pipermail/svn-src-all/2010-October/030466.html > > And I was wondering if this perhaps means that log device removal is now > possible on 8-STABLE. Or was this just a small code revision toward that > feature? > > Also I apologize in advance if my mailer does something weird and sends this > message more than once, please be sure to tell me. > Hello, This only adds the ability to offline a log device, but not to remove it. Also pools with missing log devices can't be imported. These features are added in ZFS v19. Regards, Nikolay___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
'zfs send -i': destination has been modified
I am trying to do a 'zfs send -i' and failing. This is my simple proof of concept test: Create the data # zfs create storage/a # touch /storage/a/1 # touch /storage/a/2 # touch /storage/a/3 Snapshot # zfs snapshot storage/a...@2010.10.19 send # zfs send storage/a...@2010.10.19 | zfs receive -v storage/compressed/a receiving full stream of storage/a...@2010.10.19 into storage/compressed/a...@2010.10.19 received 252KB stream in 2 seconds (126KB/sec) # Create one more file and snapshot that # touch /storage/a/4 # zfs snapshot storage/a...@2010.10.20 send it # zfs send -i storage/a...@2010.10.19 storage/a...@2010.10.20 | zfs receive -v storage/compressed/a receiving incremental stream of storage/a...@2010.10.20 into storage/compressed/a...@2010.10.20 received 250KB stream in 3 seconds (83.4KB/sec) What do we have? # find /storage/compressed/a /storage/compressed/a /storage/compressed/a/1 /storage/compressed/a/2 /storage/compressed/a/3 /storage/compressed/a/4 Of note: * FreeBSD 8.1-STABLE * ZFS filesystem version 4. * ZFS pool version 15. * zfs send is on compression off * zfs receive has compression on What I actually want to do and what fails: # zfs snapshot storage/bac...@2010.10.19 # zfs send storage/bac...@2010.10.19 | zfs receive -v storage/compressed/bacula receiving full stream of storage/bac...@2010.10.19 into storage/compressed/bac...@2010.10.19 received 4.38TB stream in 42490 seconds (108MB/sec) # zfs snapshot storage/bac...@2010.10.20 # zfs send -i storage/bac...@2010.10.19 storage/bac...@2010.10.20 | zfs receive -v storage/compressed/bacula receiving incremental stream of storage/bac...@2010.10.20 into storage/compressed/bac...@2010.10.20 cannot receive incremental stream: destination storage/compressed/bacula has been modified since most recent snapshot warning: cannot send 'storage/bac...@2010.10.20': Broken pipe I have no idea why this fails. Clues please? To my knowledge, the destination has not been written to. -- Dan Langille -- http://langille.org/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 'zfs send -i': destination has been modified
On Wed, Oct 20, 2010 at 08:32:33AM -0400, Dan Langille typed: > I am trying to do a 'zfs send -i' and failing. > > This is my simple proof of concept test: > > Create the data > # zfs create storage/a > # touch /storage/a/1 > # touch /storage/a/2 > # touch /storage/a/3 > > Snapshot > # zfs snapshot storage/a...@2010.10.19 > > send > # zfs send storage/a...@2010.10.19 | zfs receive -v storage/compressed/a > receiving full stream of storage/a...@2010.10.19 into > storage/compressed/a...@2010.10.19 > received 252KB stream in 2 seconds (126KB/sec) > # > > > > Create one more file and snapshot that > > # touch /storage/a/4 > # zfs snapshot storage/a...@2010.10.20 > > > send it > > # zfs send -i storage/a...@2010.10.19 storage/a...@2010.10.20 | zfs receive > -v > storage/compressed/a > receiving incremental stream of storage/a...@2010.10.20 into > storage/compressed/a...@2010.10.20 > received 250KB stream in 3 seconds (83.4KB/sec) > > What do we have? > > # find /storage/compressed/a > /storage/compressed/a > /storage/compressed/a/1 > /storage/compressed/a/2 > /storage/compressed/a/3 > /storage/compressed/a/4 > > > Of note: > > * FreeBSD 8.1-STABLE > * ZFS filesystem version 4. > * ZFS pool version 15. > * zfs send is on compression off > * zfs receive has compression on > > What I actually want to do and what fails: > > > # zfs snapshot storage/bac...@2010.10.19 > > # zfs send storage/bac...@2010.10.19 | zfs receive -v > storage/compressed/bacula > receiving full stream of storage/bac...@2010.10.19 into > storage/compressed/bac...@2010.10.19 > received 4.38TB stream in 42490 seconds (108MB/sec) > > # zfs snapshot storage/bac...@2010.10.20 > > # zfs send -i storage/bac...@2010.10.19 storage/bac...@2010.10.20 | zfs > receive -v storage/compressed/bacula > receiving incremental stream of storage/bac...@2010.10.20 into > storage/compressed/bac...@2010.10.20 > cannot receive incremental stream: destination storage/compressed/bacula > has been modified > since most recent snapshot > warning: cannot send 'storage/bac...@2010.10.20': Broken pipe > > I have no idea why this fails. Clues please? > > To my knowledge, the destination has not been written to. Has any read operation been done on the destination (ie: updated atime) ? Ruben ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 'zfs send -i': destination has been modified
On Wed, October 20, 2010 8:44 am, Ruben de Groot wrote: > On Wed, Oct 20, 2010 at 08:32:33AM -0400, Dan Langille typed: >> I am trying to do a 'zfs send -i' and failing. >> >> This is my simple proof of concept test: >> >> Create the data >> # zfs create storage/a >> # touch /storage/a/1 >> # touch /storage/a/2 >> # touch /storage/a/3 >> >> Snapshot >> # zfs snapshot storage/a...@2010.10.19 >> >> send >> # zfs send storage/a...@2010.10.19 | zfs receive -v storage/compressed/a >> receiving full stream of storage/a...@2010.10.19 into >> storage/compressed/a...@2010.10.19 >> received 252KB stream in 2 seconds (126KB/sec) >> # >> >> >> >> Create one more file and snapshot that >> >> # touch /storage/a/4 >> # zfs snapshot storage/a...@2010.10.20 >> >> >> send it >> >> # zfs send -i storage/a...@2010.10.19 storage/a...@2010.10.20 | zfs receive >> -v >> storage/compressed/a >> receiving incremental stream of storage/a...@2010.10.20 into >> storage/compressed/a...@2010.10.20 >> received 250KB stream in 3 seconds (83.4KB/sec) >> >> What do we have? >> >> # find /storage/compressed/a >> /storage/compressed/a >> /storage/compressed/a/1 >> /storage/compressed/a/2 >> /storage/compressed/a/3 >> /storage/compressed/a/4 >> >> >> Of note: >> >> * FreeBSD 8.1-STABLE >> * ZFS filesystem version 4. >> * ZFS pool version 15. >> * zfs send is on compression off >> * zfs receive has compression on >> >> What I actually want to do and what fails: >> >> >> # zfs snapshot storage/bac...@2010.10.19 >> >> # zfs send storage/bac...@2010.10.19 | zfs receive -v >> storage/compressed/bacula >> receiving full stream of storage/bac...@2010.10.19 into >> storage/compressed/bac...@2010.10.19 >> received 4.38TB stream in 42490 seconds (108MB/sec) >> >> # zfs snapshot storage/bac...@2010.10.20 >> >> # zfs send -i storage/bac...@2010.10.19 storage/bac...@2010.10.20 | zfs >> receive -v storage/compressed/bacula >> receiving incremental stream of storage/bac...@2010.10.20 into >> storage/compressed/bac...@2010.10.20 >> cannot receive incremental stream: destination storage/compressed/bacula >> has been modified >> since most recent snapshot >> warning: cannot send 'storage/bac...@2010.10.20': Broken pipe >> >> I have no idea why this fails. Clues please? >> >> To my knowledge, the destination has not been written to. > > Has any read operation been done on the destination (ie: updated atime) ? Not that I know of. But I do think that is the issue. Thank you. Adding a -F option to the receive helps: # zfs send -i storage/bac...@2010.10.19 storage/bac...@2010.10.20 | zfs receive -vF storage/compressed/bacula receiving incremental stream of storage/bac...@2010.10.20 into storage/compressed/bac...@2010.10.20 received 20.0GB stream in 303 seconds (67.5MB/sec) Just after I sent my email, I found this post: http://lists.freebsd.org/pipermail/freebsd-current/2007-July/075774.html Problem solved. :) -- Dan Langille -- http://langille.org/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 'zfs send -i': destination has been modified
On 2010-10-20, at 8:54 AM, Dan Langille wrote: > Not that I know of. But I do think that is the issue. Thank you. Adding > a -F option to the receive helps: > > # zfs send -i storage/bac...@2010.10.19 storage/bac...@2010.10.20 | zfs > receive -vF storage/compressed/bacula > receiving incremental stream of storage/bac...@2010.10.20 into > storage/compressed/bac...@2010.10.20 > received 20.0GB stream in 303 seconds (67.5MB/sec) > > Just after I sent my email, I found this post: > > http://lists.freebsd.org/pipermail/freebsd-current/2007-July/075774.html > > Problem solved. :) Good workaround, forcing the receive, but I'd still like to figure out what's causing the filesystem to be modified on the receiving end. But this will be useful if I can't determine what is causing it to differ. Thanks, Derek ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 'zfs send -i': destination has been modified
Seeing similar here Dan, my destination filesystems are becoming modified sometime after the previous snapshot has been sent so the incremental fails to be received. However, the server I'm sending to is not in use so I can't explain why it's changing. When I run a zdiff on the receiving system's filesystem, it can't find any changes in the file, but the byte difference between the snapshot and current can range from anywhere from 32KB to 101MB so far in my testing. If I rollback the filesystem to the last received snapshot it works, but I have no explanation for it changing. I disabled everything that was running on the receiving system yesterday afternoon and there's been no change since, so now I need to step through and see if I can figure out what process was causing the byte difference on the filesystems. Derek On 2010-10-20, at 8:32 AM, Dan Langille wrote: > I am trying to do a 'zfs send -i' and failing. > > This is my simple proof of concept test: > > Create the data > # zfs create storage/a > # touch /storage/a/1 > # touch /storage/a/2 > # touch /storage/a/3 > > Snapshot > # zfs snapshot storage/a...@2010.10.19 > > send > # zfs send storage/a...@2010.10.19 | zfs receive -v storage/compressed/a > receiving full stream of storage/a...@2010.10.19 into > storage/compressed/a...@2010.10.19 > received 252KB stream in 2 seconds (126KB/sec) > # > > > > Create one more file and snapshot that > > # touch /storage/a/4 > # zfs snapshot storage/a...@2010.10.20 > > > send it > > # zfs send -i storage/a...@2010.10.19 storage/a...@2010.10.20 | zfs receive > -v > storage/compressed/a > receiving incremental stream of storage/a...@2010.10.20 into > storage/compressed/a...@2010.10.20 > received 250KB stream in 3 seconds (83.4KB/sec) > > What do we have? > > # find /storage/compressed/a > /storage/compressed/a > /storage/compressed/a/1 > /storage/compressed/a/2 > /storage/compressed/a/3 > /storage/compressed/a/4 > > > Of note: > > * FreeBSD 8.1-STABLE > * ZFS filesystem version 4. > * ZFS pool version 15. > * zfs send is on compression off > * zfs receive has compression on > > What I actually want to do and what fails: > > > # zfs snapshot storage/bac...@2010.10.19 > > # zfs send storage/bac...@2010.10.19 | zfs receive -v > storage/compressed/bacula > receiving full stream of storage/bac...@2010.10.19 into > storage/compressed/bac...@2010.10.19 > received 4.38TB stream in 42490 seconds (108MB/sec) > > # zfs snapshot storage/bac...@2010.10.20 > > # zfs send -i storage/bac...@2010.10.19 storage/bac...@2010.10.20 | zfs > receive -v storage/compressed/bacula > receiving incremental stream of storage/bac...@2010.10.20 into > storage/compressed/bac...@2010.10.20 > cannot receive incremental stream: destination storage/compressed/bacula > has been modified > since most recent snapshot > warning: cannot send 'storage/bac...@2010.10.20': Broken pipe > > I have no idea why this fails. Clues please? > > To my knowledge, the destination has not been written to. > > > -- > Dan Langille -- http://langille.org/ > > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
latest -stable: still waiting after ...
hi, with the latest -stable, the boot process gets stuck with ... ugen2.2: at usbus2 uhub6: on usbus2 uhub6: 3 ports with 3 removable, self powered ugen3.2: at usbus3 ukbd0: on usbus3 kbd2 at ukbd0 ums0: on usbus3 ums0: 3 buttons and [Z] coordinates ID=0 <- stuck here run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #3 Launched! this does not happen with and older -stable (August) kernel Cheers, danny ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Has anyone usd hast in production yet - opinions ?
> Being the author of many problem reports I can say that most of them were not > critical and for marginal cases (like some issues with hooks or a race that > showed up when changing HAST role in loop -- you would never do this in > production). And fixes were committed in several days after a report. I don't > know any open issue. Ah, thanks for this. I have just updated everything to the latest -STABLE, and am thinking of shifting to hast sometime next week. cheers, -pete. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Spurious reboot in 8.1-RELEASE when reading from ZFS pool with > 9 disks
Hi folks, I've been playing with ZFS in 8.1-RELEASE (amd64) on a Sun Fire X4500 with 16 GB RAM and in general it seems to work well when used as recommended. BUT... In spite of the suggestion of Sun and FreeBSD developers to the contrary, I have been trying to create some zraid pools of size greater than 9 disks and it seems to give some trouble. If I try to create a pool containing more than 9 [500 GB] disks, doesn't matter if it is zraid1 or zraid2, the system seems to reboot given any amount of sustained reads from the pool (haven't tested mirrored pools). Now, I am not sure of: - Whether the reboot in the zraid1 case is caused by exactly the same issue as the reboot in the zraid2 case - Whether this is an issue of total number of member disks, or total amount of disk space in the pool. All I have to work with at the moment is 500 GB drives. I am not doing any sysctl tuning; just running with the defaults or what the system automatically sizes. I tried playing around with tuning some sysctl parameters including setting arc_max to be very small and it didn't seem to help any; pools of greater than 9 disks in size are always unstable. Writes seem to work OK; I can, say, pull stuff from over the network and save it to the pool, or I can do something like, dd if=/dev/random of=/mybigpool/bigfile bs=1024 size=10M and it will write data all day pretty happily. But if I try to read back from the pool, for example, dd if=/mybigpool/bigfile of=/dev/null bs=1024 or even to just do something like, cp /mybigpool/bigfile /mybigpool/bigfile_2 the system reboots pretty much immediately. I never see anything on the console at all; it just reboots. Even if I build a new kernel with debugging options: options KDB options DDB the system still just reboots; I never see anything on the console and I never get to the debugger. So, as I say, very easy to reproduce the problem, just create a zraid pool of any type with more than 9 member disks, dump some data to it, then try to read it back, and the machine will reboot. If I create a pool with 9 or fewer disks, the system seems perfectly stable. I was never able to reproduce the reboot behavior as long as the pools contained 9 or fewer drives, beating on it fairly hard with iozone and multiple current dd operations writing large files to and from memory. Just wondering if anyone's seen this problem before and as to whether or not it is a known bug and may have been fixed in STABLE or CURRENT? Should I report this as a bug? Should I just create pools of 9 or fewer drives? Not sure if my customer is going to want to use STABLE or CURRENT in production but I wanted to run this by the list just to see. Best, -Sean ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: Spurious reboot in 8.1-RELEASE when reading from ZFS pool with > 9 disks
Hi Lawrence, Interesting; have you tried this for raidz2 as well? I just created a raidz2 pool with 5 disks and then added another 5 disk raidz2 to it, so, total of 10 disks in the pool (though this is ultimately a losing strategy unless the number of disks is >> 9 because two drives are lost for parity in each sub-raid in the pool). It (seemed) just slightly more stable than creating a single raidz2 pool with > 9 disks but it still crashes. I guess this does allow me to say its more an issue of number of devices in the pool versus capacity of the pool because with the parity drives taken out, the pool with two 5-disk raidz2s has less total capacity than a pool with a single 9-disk raidz2. Just out of idle curiousity, I also tried it with raidz1 on my system. Again, I created a 5-disk pool, raidz1 this time, then added another 5-disk raidz1 to the pool for, again, total of 10 disks. Again, a bit of a losing strategy versus creating one great big raidz unless the number of disks is >> 9 because of losing a disk in each sub-raidz1 in the pool for parity but less so of course than raidz2. This seemed to crash too, same behavior. Are you using 8.1-RELEASE or STABLE or ...? Best, -Sean I have a 16 disk pool, if you create it with zpool create poolname raidz disk1 disk2 disk3 etc then zpool add poolname raidz disk8 disk9 disk10 etc You get the full size pool and no issues. pool: tank state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Wed Oct 20 14:54:08 2010 config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10ONLINE 0 0 0 da11ONLINE 0 0 0 da12ONLINE 0 0 0 da13ONLINE 0 0 0 da14ONLINE 0 0 0 da15ONLINE 0 0 0 errors: No known data errors ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Spurious reboot in 8.1-RELEASE when reading from ZFS pool with > 9 disks
Ahoy. I just thought I'd add a data point to the mix. I have an 11-disk v13 pool comprised of 400-GB disks on an 8.1 amd64 system and the machine behaves just fine with it: # zpool status pool: archive state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: resilver completed after 0h0m with 0 errors on Fri Oct 8 17:56:52 2010 config: NAMESTATE READ WRITE CKSUM archive ONLINE 0 0 0 raidz1ONLINE 0 0 0 ad4 ONLINE 0 0 0 133K resilvered ad6 ONLINE 0 0 0 84K resilvered ad8 ONLINE 0 0 0 85.5K resilvered ad10ONLINE 0 0 0 84.5K resilvered ad12ONLINE 0 0 0 88K resilvered ad14ONLINE 0 0 0 83.5K resilvered ad16ONLINE 0 0 0 83K resilvered ad18ONLINE 0 0 0 84.5K resilvered ad20ONLINE 0 0 0 85.5K resilvered ad22ONLINE 0 0 0 84K resilvered ad24ONLINE 0 0 0 86.5K resilvered errors: No known data errors -Boris On 10/20/10 13:26, Sean Thomas Caron wrote: Hi Lawrence, Interesting; have you tried this for raidz2 as well? I just created a raidz2 pool with 5 disks and then added another 5 disk raidz2 to it, so, total of 10 disks in the pool (though this is ultimately a losing strategy unless the number of disks is >> 9 because two drives are lost for parity in each sub-raid in the pool). It (seemed) just slightly more stable than creating a single raidz2 pool with > 9 disks but it still crashes. I guess this does allow me to say its more an issue of number of devices in the pool versus capacity of the pool because with the parity drives taken out, the pool with two 5-disk raidz2s has less total capacity than a pool with a single 9-disk raidz2. Just out of idle curiousity, I also tried it with raidz1 on my system. Again, I created a 5-disk pool, raidz1 this time, then added another 5-disk raidz1 to the pool for, again, total of 10 disks. Again, a bit of a losing strategy versus creating one great big raidz unless the number of disks is >> 9 because of losing a disk in each sub-raidz1 in the pool for parity but less so of course than raidz2. This seemed to crash too, same behavior. Are you using 8.1-RELEASE or STABLE or ...? Best, -Sean I have a 16 disk pool, if you create it with zpool create poolname raidz disk1 disk2 disk3 etc then zpool add poolname raidz disk8 disk9 disk10 etc You get the full size pool and no issues. pool: tank state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Wed Oct 20 14:54:08 2010 config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10ONLINE 0 0 0 da11ONLINE 0 0 0 da12ONLINE 0 0 0 da13ONLINE 0 0 0 da14ONLINE 0 0 0 da15ONLINE 0 0 0 errors: No known data errors ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Spurious reboot in 8.1-RELEASE when reading from ZFS pool with > 9 disks
On Wed, Oct 20, 2010 at 11:27:38AM -0400, Sean Thomas Caron wrote: > I've been playing with ZFS in 8.1-RELEASE (amd64) on a Sun Fire > X4500 with 16 GB RAM and in general it seems to work well when used > as recommended. > > BUT... > > In spite of the suggestion of Sun and FreeBSD developers to the > contrary, I have been trying to create some zraid pools of size > greater than 9 disks and it seems to give some trouble. > > If I try to create a pool containing more than 9 [500 GB] disks, > doesn't matter if it is zraid1 or zraid2, the system seems to reboot > given any amount of sustained reads from the pool (haven't tested > mirrored pools). Now, I am not sure of: > > - Whether the reboot in the zraid1 case is caused by exactly the > same issue as the reboot in the zraid2 case > > - Whether this is an issue of total number of member disks, or total > amount of disk space in the pool. All I have to work with at the > moment is 500 GB drives. > > I am not doing any sysctl tuning; just running with the defaults or > what the system automatically sizes. I tried playing around with > tuning some sysctl parameters including setting arc_max to be very > small and it didn't seem to help any; pools of greater than 9 disks > in size are always unstable. > > Writes seem to work OK; I can, say, pull stuff from over the network > and save it to the pool, or I can do something like, > > dd if=/dev/random of=/mybigpool/bigfile bs=1024 size=10M > > and it will write data all day pretty happily. But if I try to read > back from the pool, for example, > > dd if=/mybigpool/bigfile of=/dev/null bs=1024 > > or even to just do something like, > > cp /mybigpool/bigfile /mybigpool/bigfile_2 > > the system reboots pretty much immediately. I never see anything on > the console at all; it just reboots. > > Even if I build a new kernel with debugging options: > > options KDB > options DDB > > the system still just reboots; I never see anything on the console > and I never get to the debugger. > > So, as I say, very easy to reproduce the problem, just create a > zraid pool of any type with more than 9 member disks, dump some data > to it, then try to read it back, and the machine will reboot. > > If I create a pool with 9 or fewer disks, the system seems perfectly > stable. I was never able to reproduce the reboot behavior as long as > the pools contained 9 or fewer drives, beating on it fairly hard > with iozone and multiple current dd operations writing large files > to and from memory. > > Just wondering if anyone's seen this problem before and as to > whether or not it is a known bug and may have been fixed in STABLE > or CURRENT? Should I report this as a bug? Should I just create > pools of 9 or fewer drives? Not sure if my customer is going to want > to use STABLE or CURRENT in production but I wanted to run this by > the list just to see. There are users here using FreeBSD ZFS with *lots* of disks (I think someone was using 32 disks at one point) reliably. Some of them post here regularly (with other issues that don't consist of sporadic reboots). The kernel options may not be sufficient. I'm used to using these: # Debugging options options BREAK_TO_DEBUGGER # Sending a serial BREAK drops to DDB options KDB # Enable kernel debugger support options KDB_TRACE # Print stack trace automatically on panic options DDB # Support DDB options GDB # Support remote GDB And in /etc/rc.conf, setting: ddb_enable="yes" Next: arc_max isn't "technically" a sysctl, meaning it can't be changed in real-time, so I'm not sure how you managed to do that. Validation: sysctl: oid 'vfs.zfs.arc_max' is a read only tunable sysctl: Tunable values are set in /boot/loader.conf Your system may be reporting something relating to kmem exhaustion but is then auto-rebooting so fast that you can't see the message on VGA console. Do you have serial console? Please try setting the following tunables in /boot/loader.conf and reboot the machine, then see if the same problem persists. vm.kmem_size="16384M" vfs.zfs.arc_max="14336M" vfs.zfs.prefetch_disable="1" vfs.zfs.zio.use_uma="0" vfs.zfs.txg.timeout="5" I would also advocate you try 8.1-STABLE as there have been many changes in ZFS since then (and I'm not just referring to the v15 import), including how the ARC gets sized/adjusted. CURRENT is highly bleeding-edge, so I would start or stick with STABLE. Finally, there's always the possibility that the PSU has some sort of load problem with that many disks all being accessed at the same time. I imagine the power draw of that system is quite high. I can't imagine Sun shipping a box with a insufficient PSU, but then again power draw changes depending on the RPM of the disks used and many other things. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking
Re: kpanic on install >32GB of RAM [SEC=UNCLASSIFIED]
on 20/10/2010 21:05 Sean Bruno said the following: > HP testers report that 7 nor 8 install without the kernel > 'insta-panicing' across all CPUS at the same time. > > I'm going to ask that they try the latest 9 snap shot and then kill the > project at this time. So you are not willing to try the patch? -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: kpanic on install >32GB of RAM [SEC=UNCLASSIFIED]
On Tue, 2010-10-19 at 06:01 -0700, John Baldwin wrote: > On Tuesday, October 19, 2010 1:16:07 am Andriy Gapon wrote: > > on 19/10/2010 06:11 Wilkinson, Alex said the following: > > > Will it work something like http://freshmeat.net/projects/mcelog/ ? > > > > jhb has a port of this, yes. > > The relevant bits are at http://www.FreeBSD.org/~jhb/mcelog/ > > I should probably make this into a port, but I really need to e-mail the > mcelog folks about integrating the patches into their tree. > Q: Has anyone installed any version of FreeBSD on a DL980 G7 from HP yet? Has anyone installed any version of FreeBSD on a DL580 G7 from HP? HP testers report that 7 nor 8 install without the kernel 'insta-panicing' across all CPUS at the same time. I'm going to ask that they try the latest 9 snap shot and then kill the project at this time. sean ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: kpanic on install >32GB of RAM [SEC=UNCLASSIFIED]
on 20/10/2010 21:16 Sean Bruno said the following: > Huh ... did I miss it? I'm willing to try a lot. Perhaps then :-) This was my post: http://thread.gmane.org/gmane.os.freebsd.stable/72450/focus=72515 It had a link to a parallel discussion on current: http://thread.gmane.org/gmane.os.freebsd.current/128099/focus=128576 Which had a link to: http://people.freebsd.org/~avg/uma-many-cpus.diff Which is the patch :) -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: kpanic on install >32GB of RAM [SEC=UNCLASSIFIED]
On Wed, 2010-10-20 at 11:11 -0700, Andriy Gapon wrote: > on 20/10/2010 21:05 Sean Bruno said the following: > > HP testers report that 7 nor 8 install without the kernel > > 'insta-panicing' across all CPUS at the same time. > > > > I'm going to ask that they try the latest 9 snap shot and then kill the > > project at this time. > > So you are not willing to try the patch? > Huh ... did I miss it? I'm willing to try a lot. sean ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: kpanic on install >32GB of RAM [SEC=UNCLASSIFIED]
On Wed, 2010-10-20 at 11:20 -0700, Andriy Gapon wrote: > on 20/10/2010 21:16 Sean Bruno said the following: > > Huh ... did I miss it? I'm willing to try a lot. > > Perhaps then :-) > This was my post: > http://thread.gmane.org/gmane.os.freebsd.stable/72450/focus=72515 > > It had a link to a parallel discussion on current: > http://thread.gmane.org/gmane.os.freebsd.current/128099/focus=128576 > > Which had a link to: > http://people.freebsd.org/~avg/uma-many-cpus.diff > > Which is the patch :) So here's the problem. Machine in in Houston, can't netboot/nfsboot it. The basic installers all will get me to the Beastie loader, but fail after loading the kernel and attempting to boot. I guess, I could replace the kernel on the CD and have them reburn it? Sean ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: kpanic on install >32GB of RAM [SEC=UNCLASSIFIED]
On Wed, 20 Oct 2010 11:28:21 -0700 Sean Bruno wrote: > So here's the problem. > Machine in in Houston, can't netboot/nfsboot it. The basic installers > all will get me to the Beastie loader, but fail after loading the kernel > and attempting to boot. > I guess, I could replace the kernel on the CD and have them reburn it? You may create a booting USB stick and send them the image. -- WBR, Boris Samorodov (bsam) Research Engineer, http://www.ipt.ru Telephone & Internet SP FreeBSD Committer, http://www.FreeBSD.org The Power To Serve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Spurious reboot in 8.1-RELEASE when reading from ZFS pool with > 9 disks
Hi Jeremy, Thanks for the very helpful response! I added all debugging options that you specified to my kernel and rebuilt; then set the kernel parameters as you mention (I was being a bit lazy earlier when I called them sysctls; I always tuned them in loader.conf; just that you can view their values with sysctl). Rebooted the system with the new kernel and set up a 11-disk zraid2 pool again then started beating on it. At first it seemed to be a bit more resilient with this set of kernel parameters but eventually it too failed out. Again I just got a straight up reboot, no debugger, no output to the console flashed by as far as I can tell. I don't have a serial console hooked up right now but it's probably possible to do so through the ILOM or equivalent; I will have to look into that further. This is pretty wierd. I am thinking there might be some memory starting to go in this system; never seen failing memory in an ECC box cause reboots this consistently and only under such specific conditions but I suppose it isn't completely out of the question. I'll talk to my customer and see what they can do about the hardware; maybe they have some spares. I will also try 8.1-STABLE when I have a chance and see if that works better. But it's definitely helpful to know that folks have > 9 disk raidz pools up and running on FreeBSD 8.x with no trouble - that it "should work". And the list of tunables is very useful; nice to have something to work with that I can have a bit more confidence in outside of my own guessing :) I will report back to the list when I have more information. Thanks! -Sean Quoting Jeremy Chadwick : There are users here using FreeBSD ZFS with *lots* of disks (I think someone was using 32 disks at one point) reliably. Some of them post here regularly (with other issues that don't consist of sporadic reboots). The kernel options may not be sufficient. I'm used to using these: # Debugging options options BREAK_TO_DEBUGGER # Sending a serial BREAK drops to DDB options KDB # Enable kernel debugger support options KDB_TRACE # Print stack trace automatically on panic options DDB # Support DDB options GDB # Support remote GDB And in /etc/rc.conf, setting: ddb_enable="yes" Next: arc_max isn't "technically" a sysctl, meaning it can't be changed in real-time, so I'm not sure how you managed to do that. Validation: sysctl: oid 'vfs.zfs.arc_max' is a read only tunable sysctl: Tunable values are set in /boot/loader.conf Your system may be reporting something relating to kmem exhaustion but is then auto-rebooting so fast that you can't see the message on VGA console. Do you have serial console? Please try setting the following tunables in /boot/loader.conf and reboot the machine, then see if the same problem persists. vm.kmem_size="16384M" vfs.zfs.arc_max="14336M" vfs.zfs.prefetch_disable="1" vfs.zfs.zio.use_uma="0" vfs.zfs.txg.timeout="5" I would also advocate you try 8.1-STABLE as there have been many changes in ZFS since then (and I'm not just referring to the v15 import), including how the ARC gets sized/adjusted. CURRENT is highly bleeding-edge, so I would start or stick with STABLE. Finally, there's always the possibility that the PSU has some sort of load problem with that many disks all being accessed at the same time. I imagine the power draw of that system is quite high. I can't imagine Sun shipping a box with a insufficient PSU, but then again power draw changes depending on the RPM of the disks used and many other things. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
