Re: [zfs-discuss] confusion and frustration with zpool
Just to close the loop on this, for some other poor soul having similar problems and googling away I believe I have resolved it. The problem was somewhere on the 750G drive, and was fixed by detaching and re-attaching it to my mirrors. I actually took the extra step of creating a UFS on the largest slice of the 750G and copying the data, with the thought that I may not be able to get my data back from ZFS, but after detaching the disk and doing a scrub, there was only one more data error with another MP3 that's easily replaced, and the filesystem was clean. Re-attaching the 750G drive's slices to the clean filesystem has not resulted in any further problems in something over a week so far. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] confusion and frustration with zpool
I have a zpool which has grown "organically". I had a 60Gb disk, I added a 120, I added a 500, I got a 750 and sliced it and mirrored the other pieces. The 60 and the 120 are internal PATA drives, the 500 and 750 are Maxtor OneTouch USB drives. The original system I created the 60+120+500 pool on was Solaris 10 update 3, patched to use ZFS sometime last fall (November I believe). In early June, a storm blew out my root drive. Thinking it was an opportunity to upgrade, I re-installed with OpenSolaris, and completed the mirroring which I had intended for some time, and upgraded zfs from v4 to v10. The system was not stable. Reading around, I realized that 512M of RAM and a 32-bit CPU was probably a poor choice for an OpenSolaris, ZFS based web and file server for my home. So I purchased an ASUS AMD64x2 system and 4G of RAM and this weekend I was able to get that set up. However, my pool is not behaving well. I have had "insufficient replicas" for the pool and "corrupted data" for the mirror piece that is on both the USB drives. This confuses me because I'm also seeing "no known data errors" which leads me to wonder where this corrupted data might be. I did a zpool scrub, thinking I could shake out what the problem was; earlier when the system was unstable doing this pointed out a couple of MP3 files that were incorrect, and as they were easily replaced I just removed them and was able to get a clean filesystem. My most recent attempt to clear this involved removing the 750G drive and then trying to bring it online; this had no effect, but now the 750 is on c0 rather than c7 at the OS device level. I've googled for some guidance and found advice to export/import, and while this cleared the original insufficient replicas problem, it has not done anything for the alleged corrupted data. I have a couple thousand family photos (many of which are backed up elsewhere, but would be a huge problem to re-import) and several thousand MP3s and AACs (iTunes songs, many of which are backed up, but many are not because of being recently purchased). I've been hearing how ZFS is the way I should go, which is why I made this change last fall, but at this point I am only having confusion and frustration. Any advice for other steps I could take to recover would be great. here is some data directly from the system (yes, I know, somewhere along the line I set the date one day ahead of the real date, I will be fixing that later :) ): -bash-3.2# zpool status local pool: local state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM local DEGRADED 0 0 0 mirror ONLINE 0 0 0 c6d1p0ONLINE 0 0 0 c0t0d0s3 ONLINE 0 0 0 mirror ONLINE 0 0 0 c6d0p0ONLINE 0 0 0 c0t0d0s4 ONLINE 0 0 0 mirror UNAVAIL 0 0 0 corrupted data c8t0d0p0 ONLINE 0 0 0 c0t0d0s5 ONLINE 0 0 0 errors: No known data errors -bash-3.2# zpool history local History for 'local': 2007-11-19.11:45:11 zpool create -m /local2 local c1d0p0 2007-11-19.13:38:44 zfs recv local/main 2007-11-19.13:52:51 zfs set mountpoint=/local-pool local 2007-11-19.13:53:09 zfs set mountpoint=/local local/main 2007-11-19.14:00:48 zpool add local c1d1p0 2007-11-19.14:26:35 zfs destroy local/[EMAIL PROTECTED] 2007-11-28.18:38:26 zpool add local /dev/dsk/c3t0d0p0 2008-05-12.10:20:48 zfs set canmount=off local 2008-05-12.10:21:24 zfs set mountpoint=/ local 2008-06-16.15:56:29 zpool import -f local 2008-06-16.15:58:04 zpool export local 2008-06-27.21:41:35 zpool import local 2008-06-27.22:42:09 zpool attach -f local c5d0p0 c7t0d0s3 2008-06-28.09:06:51 zpool clear local c5d0p0 2008-06-28.09:07:00 zpool clear local c7t0d0s3 2008-06-28.09:07:11 zpool clear local 2008-06-28.09:35:39 zpool attach -f local c5d1p0 c7t0d0s4 2008-06-28.09:36:23 zpool attach -f local c6t0d0p0 c7t0d0s5 2008-06-28.13:15:26 zpool clear local 2008-06-28.13:16:48 zpool scrub local 2008-06-28.18:30:19 zpool clear local 2008-06-28.18:30:37 zpool upgrade local 2008-06-28.18:53:33 zfs create -o mountpoint=/opt/csw local/csw 2008-06-28.21:59:38 zpool export local 2008-07-06.23:25:41 zpool import local 2008-07-06.23:26:19 zpool scrub local 2008-07-07.08:40:13 zpool clear local 2008-07-07.08:43:39 zpool export local 2008-07-07.08:43:54 zpool import local 2008-07-07.08:44:20 zpool clear local 2008-07-07.08:47:20 zpool export local 2008-07-07.08:56:49 zpool import local 2008-07-07.08:58:57 zpool export local 2008-07-07.09:00:26 zpool import local 2008-07-07.09:18:16 zpool export local 2008-07-07.09:18:26 zpool import local This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.open
Re: [zfs-discuss] confusion and frustration with zpool
I'm doing another scrub after clearing "insufficient replicas" only to find that I'm back to the report of insufficient replicas, which basically leads me to expect this scrub (due to complete in about 5 hours from now) won't have any benefit either. -bash-3.2# zpool status local pool: local state: FAULTED scrub: scrub in progress for 0h32m, 9.51% done, 5h11m to go config: NAME STATE READ WRITE CKSUM local FAULTED 0 0 0 insufficient replicas mirror ONLINE 0 0 0 c6d1p0ONLINE 0 0 0 c0t0d0s3 ONLINE 0 0 0 mirror ONLINE 0 0 0 c6d0p0ONLINE 0 0 0 c0t0d0s4 ONLINE 0 0 0 mirror UNAVAIL 0 0 0 corrupted data c8t0d0p0 ONLINE 0 0 0 c0t0d0s5 ONLINE 0 0 0 errors: No known data errors This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] confusion and frustration with zpool
other6058878 Jun 10 18:51 East of Scientist Corner (II Pieces).mp3 -rwxr--r-- 1 elmegil other7244195 Jun 10 18:51 Every Dub Shall Scrub.mp3 -rwxr--r-- 1 elmegil other6878897 Jun 10 18:52 Just say Dub... Who.mp3 -rwxr--r-- 1 elmegil other8197144 Jun 10 18:51 Keep a good Dub Rubbing.mp3 -rwxr--r-- 1 elmegil other4929531 Jun 10 18:51 Satta Dread Dub.mp3 -rwxr--r-- 1 elmegil other7873642 Jun 10 18:51 Taxi to Baltimore Dub.mp3 -rwxr--r-- 1 elmegil other4438008 Jun 10 18:52 Words of Dub.mp3 -bash-3.2# rm 'Satta Dread Dub.mp3' rm: Satta Dread Dub.mp3 not removed: No space left on device Running export/import again shows data corruption again, but otherwise has the same symptom. This is strange to me because previously the other files that were corrupted didn't object to being removed. Someone else wrote me directly and suggested this could be the fault of the new hardware...but the old hardware was panicking in ZFS so it wasn't any more reliable (read: not any help to recover my data), and I half expect that the panics could be related to some of this problem too. I definitely am not seeing any other symptoms of bad hardware, no transport or other disk errors aside from the ZFS complaints (i.e. none of the usb or disk drivers are having any reported issues afaics), I'm not seeing ECC or other memory issues, no panicing from bit flips...which doesn't rule out bad hardware of course, but I think I'd expect to see more than just the ZFS problems Just as a point of information, the motherboard is an ASUS M2A-VM and I've updated to the latest available BIOS (1705 I believe it was, from March this year). I did that before the first import of the local pool on the new HW in fact. Part of me is thinking what I ought to do is lop off the 750G drive, make it its own pool, physically copy as much of the data as I can save into that pool, scrub it to be sure it's ok beyond that, and then re-create the original pool from scratch and copy the data back before mirroring again to the 750. Very drastic, seems risky. If there is anything more intelligible than I can discern from the fmdump above (fmdump -eV gives even more cryptic hex strings :) ) that could save this radical approach, any advice is appreciated. Unfortunately there aren't any other available media big enough to store 230G in a reasonable amount of time/individual media count (60 DVDs! 8G DVDs would be half that, but I have yet to find a DL drive that works reliably for me). Thanks Jeff. I hope my frustration in all this doesn't sound directed at anyone in particular and definitely not you. I appreciate your time looking and giving advice. Thanks Pete Jeff Bonwick wrote: > As a first step, 'fmdump -ev' should indicate why it's complaining > about the mirror. > > Jeff > > On Sun, Jul 06, 2008 at 07:55:22AM -0700, Pete Hartman wrote: >> I'm doing another scrub after clearing "insufficient replicas" only to find >> that I'm back to the report of insufficient replicas, which basically leads >> me to expect this scrub (due to complete in about 5 hours from now) won't >> have any benefit either. >> >> -bash-3.2# zpool status local >> pool: local >> state: FAULTED >> scrub: scrub in progress for 0h32m, 9.51% done, 5h11m to go >> config: >> >> NAME STATE READ WRITE CKSUM >> local FAULTED 0 0 0 insufficient replicas >> mirror ONLINE 0 0 0 >> c6d1p0ONLINE 0 0 0 >> c0t0d0s3 ONLINE 0 0 0 >> mirror ONLINE 0 0 0 >> c6d0p0ONLINE 0 0 0 >> c0t0d0s4 ONLINE 0 0 0 >> mirror UNAVAIL 0 0 0 corrupted data >> c8t0d0p0 ONLINE 0 0 0 >> c0t0d0s5 ONLINE 0 0 0 >> >> errors: No known data errors >> >> >> This message posted from opensolaris.org >> ___ >> zfs-discuss mailing list >> zfs-discuss@opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] confusion and frustration with zpool
I'm curious which enclosures you've had problems with? Mine are both Maxtor One Touch; the 750 is slightly different in that it has a FireWire port as well as USB. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
