Re: [zfs-discuss] Slow Resilvering Performance
i might be wrong because i'm kind of new but i THINK you need to disable automatic snapshots when resilvering, at least on the older version you did. if not it would restart every time a new snapshot was madebut then again, i may be wrong. On Fri, Jul 10, 2009 at 6:18 PM, Galen wrote: > I know this topic has been discussed many times... but what the hell makes > zpool resilvering so slow? I'm running OpenSolaris 2009.06. > > I have had a large number of problematic disks due to a bad production > batch, leading me to resilver quite a few times, progressively replacing > each disk as it dies (and now preemptively removing disks.) My complaint is > that resilvering ends up taking... days! The average write rate to the disk > being resilvered is 1 to 3 MB/sec. > > You can see zpool status and iostat -v output here: > http://pastebin.com/mcbb8dfd > > When I read files off the zpool, I get quite a few MB/sec even in a > degraded state, although the zpool is idle while resilvering in this case - > no snapshots or anything happening on it. The system has 3 GB of RAM and a > 2.8 GHz dual core CPU which is always >90% idle while resilvering. The > number of I/O operations per second is nowhere near the disk's limits. > Scrubbing takes 3-4 hours at the most, so it's clearly not a read > bottleneck. Even if I have a configuration where only one disk is being > replaced (and all others are OK), I never pass the 1-3 MB/sec limit. > > What is going on? I have had to resilver 4 times so far, and I have to > resilver at least once more. Each resilvering takes a day or two, and I cant > see why... it's not CPU, it's not sustained read throughput, it's not IOPS, > so what is it?? > > Galen > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zfs rpool boot failed
Dear Admins I had solaris 10u8 installation based on ZFS (rpool)filesystem on two mirrored scsi disks in sunfire v880. but after some months,when i reboot server with reboot command,it didnt boot from disks,and returns cant boot from boot media. how can i recover some data from my previous installation? also i run >boot disk0 (failed) >boot disk1 (failed) also run >probe-scsi-all,,then boot from each disk,it returns failed,,why? thank for any guide Regards ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] recover data after zpool create
Carson, please provide an example how to read bytes. I talk about cylinder because I don't know better. I need to read from a partition which shows as /dev/hda3 under Gparted with starting sector ending sector z. under solaris I think it becomes /dev/dsk/c0d0p3 I tried dd if=/dev/dsk/c0d0p3 ibs=256 hoping to read the first 256 bytes but it started streaming everything and beeping non-stop. even ^C could not stop the beeping. Thank you Stephen -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] recover data after zpool create
On 11 July, 2009 - stephen bond sent me these 0,6K bytes:
> Carson,
>
> please provide an example how to read bytes. I talk about cylinder because I
> don't know better.
> I need to read from a partition which shows as /dev/hda3 under Gparted with
> starting sector ending sector z.
>
> under solaris I think it becomes /dev/dsk/c0d0p3
>
> I tried
> dd if=/dev/dsk/c0d0p3 ibs=256
> hoping to read the first 256 bytes but it started streaming everything
> and beeping non-stop. even ^C could not stop the beeping.
Add of=/where/you/want/the/data unless you want the raw dump to your
screen, which is what you just got.. Add count=1 if you just want one
block.. etc.
/Tomas
--
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS tale of woe and fail
mhh, i think i`m afraid, too, as i also need to use zfs on a single, large lun. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Resilvering Loop
Galen wrote: I have a situation where my zpool (with two radiz2s) is resilvering and reaches a certain point, then starts over. There no read, write or checksum errors. The disks do have a fair amount of resilvering to do, as I've had a variety of disk failures. But at the core of things, there's enough parity data to avoid data loss, and the zpool is functioning fine, albeit in a degraded state. I've been trapped in this loop before and it's been resolved by changing the disk configuration somehow. But that is just like shooting in the dark and can be a very time-consuming process. Is there a more proper way to approach this issue? Should I be filing a bug report? What release and/or build? -- Andrew ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Resilvering Loop
I have a situation where my zpool (with two radiz2s) is resilvering and reaches a certain point, then starts over. There no read, write or checksum errors. The disks do have a fair amount of resilvering to do, as I've had a variety of disk failures. But at the core of things, there's enough parity data to avoid data loss, and the zpool is functioning fine, albeit in a degraded state. I've been trapped in this loop before and it's been resolved by changing the disk configuration somehow. But that is just like shooting in the dark and can be a very time-consuming process. Is there a more proper way to approach this issue? Should I be filing a bug report? -Galen ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Slow Resilvering Performance
On Jul 10, 2009, at 3:32 PM, Miles Nordin wrote: "g" == Galen writes: g> the disk being resilvered is 1 to 3 MB/sec. see: 6592835 resilvering is at least 10x too slow 6602697 pool has small write throughput during resilver http://bugs.opensolaris.org/view_bug.do?bug_id=6722540 -- some mention of the Dirty Time Log http://blogs.sun.com/ahrens/entry/new_scrub_code 6698575 -- zfs scrub causes multi-second lag over NFS In particular 6333409 in snv_102 is supposed to speed up scrub/resilver. I'm running OpenSolaris 2009.06, and while those are interesting to read, none of them really explain or solve my issues. -Galenh ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Resilvering Loop
I've never used snapshotting on this zpool and I can't imagine how it would snapshot without my involvement. -Galen On Jul 11, 2009, at 2:22 PM, Thomas Burgess wrote: do you have auto-snapshots on? i think old versions would restart on every single new snapshot On Sat, Jul 11, 2009 at 5:19 PM, Galen wrote: I have a situation where my zpool (with two radiz2s) is resilvering and reaches a certain point, then starts over. There no read, write or checksum errors. The disks do have a fair amount of resilvering to do, as I've had a variety of disk failures. But at the core of things, there's enough parity data to avoid data loss, and the zpool is functioning fine, albeit in a degraded state. I've been trapped in this loop before and it's been resolved by changing the disk configuration somehow. But that is just like shooting in the dark and can be a very time-consuming process. Is there a more proper way to approach this issue? Should I be filing a bug report? -Galen ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Resilvering Loop
On Jul 11, 2009, at 2:24 PM, Andrew Gabriel wrote: Galen wrote: I have a situation where my zpool (with two radiz2s) is resilvering and reaches a certain point, then starts over. There no read, write or checksum errors. The disks do have a fair amount of resilvering to do, as I've had a variety of disk failures. But at the core of things, there's enough parity data to avoid data loss, and the zpool is functioning fine, albeit in a degraded state. I've been trapped in this loop before and it's been resolved by changing the disk configuration somehow. But that is just like shooting in the dark and can be a very time-consuming process. Is there a more proper way to approach this issue? Should I be filing a bug report? What release and/or build? -- Andrew OpenSolaris 2009.06, all updates applied, pretty much clean install. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Resilvering Loop
On Jul 11, 2009, at 5:41 PM, Galen wrote: On Jul 11, 2009, at 2:24 PM, Andrew Gabriel wrote: Galen wrote: I have a situation where my zpool (with two radiz2s) is resilvering and reaches a certain point, then starts over. There no read, write or checksum errors. The disks do have a fair amount of resilvering to do, as I've had a variety of disk failures. But at the core of things, there's enough parity data to avoid data loss, and the zpool is functioning fine, albeit in a degraded state. I've been trapped in this loop before and it's been resolved by changing the disk configuration somehow. But that is just like shooting in the dark and can be a very time-consuming process. Is there a more proper way to approach this issue? Should I be filing a bug report? What release and/or build? -- Andrew OpenSolaris 2009.06, all updates applied, pretty much clean install. Did you make sure the auto snapshot service was disabled? Otherwise it will be taking hourly snapshots. I think it's enabled by default. -Ross ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] destroying large zvols interrupts pool access
I have a recurrent problem when I destroy pools, in that it disrupts access to the pool for a long period. I am using this for iscsi and FC storage. Due to numerous issues with zvols, I am migrating away from them in favor of file-based backing-stores. I would be willing to run tests to help fix this issue though. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] destroying large zvols interrupts pool access
Sorry to reply to myself, the most recent host I have seen this on is running snv_115 it's pool is running ZFS pool version 15 -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] deduplication
When it comes out, how will it work? Does it work at the pool level or a zfs file system level? If I create a zpool called 'zones' and then I create several zones underneath that, could I expect to see a lot of disk space savings if I enable dedup on the pool? Just curious as to what's coming and what I should look out for. If it works at the pool level, that'll be very useful and save quite a disk space on a system with a lot of zones - which share a lot of common files. Thanks! -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Resilvering Loop
On Jul 11, 2009, at 4:00 PM, Ross Walker wrote: On Jul 11, 2009, at 5:41 PM, Galen wrote: On Jul 11, 2009, at 2:24 PM, Andrew Gabriel wrote: Galen wrote: I have a situation where my zpool (with two radiz2s) is resilvering and reaches a certain point, then starts over. There no read, write or checksum errors. The disks do have a fair amount of resilvering to do, as I've had a variety of disk failures. But at the core of things, there's enough parity data to avoid data loss, and the zpool is functioning fine, albeit in a degraded state. I've been trapped in this loop before and it's been resolved by changing the disk configuration somehow. But that is just like shooting in the dark and can be a very time-consuming process. Is there a more proper way to approach this issue? Should I be filing a bug report? What release and/or build? -- Andrew OpenSolaris 2009.06, all updates applied, pretty much clean install. Did you make sure the auto snapshot service was disabled? Otherwise it will be taking hourly snapshots. I think it's enabled by default. -Ross There's no snapshotting happening to my knowledge. Moreover, if I change the disk configuration around enough, it will resilver without issues. -Galen ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Slow Resilvering Performance
On Jul 11, 2009, at 6:10 PM, Miles Nordin wrote: "g" == Galen writes: g> I'm running OpenSolaris 2009.06, and while those are g> interesting to read, none of them really explain or solve my g> issues. read this again: In particular 6333409 in snv_102 is supposed to speed up scrub/resilver. It is horribly slow and I am running a newer build than that - this is why I say none of them solve or explain. ga...@solaribyte:~# uname -a SunOS solaribyte 5.11 snv_111b i86pc i386 i86pc -Galen ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] deduplication
Anil wrote: Does it work at the pool level or a zfs file system level? If I create a zpool called 'zones' and then I create several zones underneath that, could I expect to see a lot of disk space savings if I enable dedup on the pool? You can get the same savings by cloning your zones. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] deduplication
On Sat, Jul 11, 2009 at 9:32 PM, Ian Collins wrote: > > Anil wrote: >> >> Does it work at the pool level or a zfs file system level? If I create a >> zpool called 'zones' and then I create several zones underneath that, could >> I expect to see a lot of disk space savings if I enable dedup on the pool? >> >> > > You can get the same savings by cloning your zones. Until you patch (S10) or image-update (OpenSolaris) or need to restore from tape. Cloning is very interesting from a speed to provision standpoint but is begging for trouble in the long term without deduplication to handle ongoing operation. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] deduplication
Anil wrote: When it comes out, how will it work? Does it work at the pool level or a zfs file system level? If I create a zpool called 'zones' and then I create several zones underneath that, could I expect to see a lot of disk space savings if I enable dedup on the pool? Just curious as to what's coming and what I should look out for. If it works at the pool level, that'll be very useful and save quite a disk space on a system with a lot of zones - which share a lot of common files. Good question - we don't know just yet. I expect we'll have a much better idea after Jeff and Bill deliver their keynote speech at Kernel Conference Australia, which kicks off on Wednesday. James C. McPherson -- Solaris kernel software engineer, system admin and troubleshooter http://www.jmcp.homeunix.com/blog http://blogs.sun.com/jmcp Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
