On Jun 16, 2012, at 10:13 AM, Scott Aitken wrote: > On Sat, Jun 16, 2012 at 09:58:40AM -0500, Gregg Wonderly wrote: >> >> On Jun 16, 2012, at 9:49 AM, Scott Aitken wrote: >> >>> On Sat, Jun 16, 2012 at 09:09:53AM -0500, Gregg Wonderly wrote: >>>> Use 'dd' to replicate as much of lofi/2 as you can onto another device, >>>> and then >>>> cable that into place? >>>> >>>> It looks like you just need to put a functioning, working, but not correct >>>> device, in that slot so that it will import and then you can 'zpool >>>> replace' the >>>> new disk into the pool perhaps? >>>> >>>> Gregg Wonderly >>>> >>>> On 6/16/2012 2:02 AM, Scott Aitken wrote: >>>>> On Sat, Jun 16, 2012 at 08:54:05AM +0200, Stefan Ring wrote: >>>>>>> when you say remove the device, I assume you mean simply make it >>>>>>> unavailable >>>>>>> for import (I can't remove it from the vdev). >>>>>> Yes, that's what I meant. >>>>>> >>>>>>> root@openindiana-01:/mnt# zpool import -d /dev/lofi >>>>>>> ??pool: ZP-8T-RZ1-01 >>>>>>> ?? ??id: 9952605666247778346 >>>>>>> ??state: FAULTED >>>>>>> status: One or more devices are missing from the system. >>>>>>> action: The pool cannot be imported. Attach the missing >>>>>>> ?? ?? ?? ??devices and try again. >>>>>>> ?? see: http://www.sun.com/msg/ZFS-8000-3C >>>>>>> config: >>>>>>> >>>>>>> ?? ?? ?? ??ZP-8T-RZ1-01 ?? ?? ?? ?? ?? ?? ??FAULTED ??corrupted data >>>>>>> ?? ?? ?? ?? ??raidz1-0 ?? ?? ?? ?? ?? ?? ?? ??DEGRADED >>>>>>> ?? ?? ?? ?? ?? ??12339070507640025002 ??UNAVAIL ??cannot open >>>>>>> ?? ?? ?? ?? ?? ??/dev/lofi/5 ?? ?? ?? ?? ?? ONLINE >>>>>>> ?? ?? ?? ?? ?? ??/dev/lofi/4 ?? ?? ?? ?? ?? ONLINE >>>>>>> ?? ?? ?? ?? ?? ??/dev/lofi/3 ?? ?? ?? ?? ?? ONLINE >>>>>>> ?? ?? ?? ?? ?? ??/dev/lofi/1 ?? ?? ?? ?? ?? ONLINE >>>>>>> >>>>>>> It's interesting that even though 4 of the 5 disks are available, it >>>>>>> still >>>>>>> can import it as DEGRADED. >>>>>> I agree that it's "interesting". Now someone really knowledgable will >>>>>> need to have a look at this. I can only imagine that somehow the >>>>>> devices contain data from different points in time, and that it's too >>>>>> far apart for the aggressive txg rollback that was added in PSARC >>>>>> 2009/479. Btw, did you try that? Try: zpool import -d /dev/lofi -FVX >>>>>> ZP-8T-RZ1-01. >>>>>> >>>>> Hi again, >>>>> >>>>> that got slightly further, but still no dice: >>>>> >>>>> root@openindiana-01:/mnt# zpool import -d /dev/lofi -FVX ZP-8T-RZ1-01 >>>>> root@openindiana-01:/mnt# zpool list >>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>> ZP-8T-RZ1-01 - - - - - FAULTED - >>>>> rpool 15.9G 2.17G 13.7G 13% 1.00x ONLINE - >>>>> root@openindiana-01:/mnt# zpool status >>>>> pool: ZP-8T-RZ1-01 >>>>> state: FAULTED >>>>> status: One or more devices could not be used because the label is missing >>>>> or invalid. There are insufficient replicas for the pool to >>>>> continue >>>>> functioning. >>>>> action: Destroy and re-create the pool from >>>>> a backup source. >>>>> see: http://www.sun.com/msg/ZFS-8000-5E >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> ZP-8T-RZ1-01 FAULTED 0 0 1 corrupted >>>>> data >>>>> raidz1-0 ONLINE 0 0 6 >>>>> 12339070507640025002 UNAVAIL 0 0 0 was >>>>> /dev/lofi/2 >>>>> /dev/lofi/5 ONLINE 0 0 0 >>>>> /dev/lofi/4 ONLINE 0 0 0 >>>>> /dev/lofi/3 ONLINE 0 0 0 >>>>> /dev/lofi/1 ONLINE 0 0 0 >>>>> >>>>> root@openindiana-01:/mnt# zpool scrub ZP-8T-RZ1-01 >>>>> cannot scrub 'ZP-8T-RZ1-01': pool is currently unavailable >>>>> >>>>> Thanks for your tenacity Stefan. >>>>> Scott >>>>> _______________________________________________ >>>>> zfs-discuss mailing list >>>>> zfs-discuss@opensolaris.org >>>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>>>> >>>> >>>> >>> >>> Hi Greg, >>> >>> lofi/2 is a dd of a real disk. I am using disk images because I can roll >>> back, clone etc without using the original drives (which are long gone >>> anyway). >>> >>> I have tried making /2 unavailable for import, and zfs just moans that it >>> can't be opened. It fails to import even though I have only one disk >>> missing >>> of a RAIDZ array. >> >> My experience is that ZFS will not import a pool with a missing disk. There >> has to be something in that slot before the import will occur. Even if the >> disk is "corrupt", it needs to be there. I think this is a failsafe >> mechanism that tries to keep a pool from going live when you have mistakenly >> not connected all the drives. That keeps the disks from becoming >> chronologically/txn misaligned which can result in data loss, in the right >> combinations I believe. >> >> Gregg Wonderly >> > > Hi again Gregg, > > not sure if I should be top posting this... > > Given I am working with images, it's hard to put just anything "in place" of > lofi/2. ZFS scans all of the files in the directory for ZFS labels, so just > replacing lofi/2 with an empty file (for example) just means ZFS skips it, > which is the same result as deleting lofi/2 altogether. I did this, but to > no avail. ZFS complains about having insufficient replicas.
I don't really know much about the total space layout of a ZFS disk surface, because I haven't had to do something like this with ZFS. But, I have done something like this for other files where I needed to replace something on the end that was corrupt with something that made the beginning usable. The 3 commands below take some space off of lori/2 and the rest off of lofi/3 and create a new set of bits that you can try as lofi/2. # make sure you have a copy of lofi/2 somewhere else that you are keeping safe, it seems you do, then: dd if=lofi/3 of=end-of-lofi-2 skip=10 bs=102400 dd if=lofi/2 of=front-of-lofi-2 count=10 bs=102400 cat front-of-lofi-2 end-of-lofi-2 >lofi/2 The cannot open message likely just means some corruption has occurred as you are saying. All I think you need is some disk image that starts to look like lofi/2 should look, but in the end you don't care about lofi/2 being correct, as long as the other disks work sp that you can resilver a replacement, or at least import and copy the data. My experience has been that getting something that is readable there is all that is needed. It will then tell you that the disk is "corrupt" instead of "unavail" and the pool should then do something different, if not be accessible. Gregg > There is something more going on here which is beyond my capabilities. Given > the history I outlined earlier, there is almost certainly one or two corrupt > labels on those disks (images). > > Scott _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
