On 2012-Oct-31 17:25:09 -0000, Steven Hartland <ste...@multiplay.co.uk> wrote:
Been running some tests on new hardware here to verify all
is good. One of the tests was to fill the zfs array which
seems like its totally corrupted the tank.
I've accidently "filled" a pool, and had multiple processes try to
write to the full pool, without either emptying the free space reserve
(so I could still delete the offending files) or corrupting the pool.
Same here but its the first time I've had ZIL in place at the time so
wondering if that may be playing a factor.
Had you tried to read/write the raw disks before you tried the
ZFS testing?
Yes, didn't see any issues but then it wasn't checksuming so tbh I
wouldn't have noticed if it was silently corrupting data.
Do you have compression and/or dedupe enabled on the pool?
Nope bog standard raidz2 no additional settings
1. Given the information it seems like the multiple writes filling
the disk may have caused metadata corruption?
I don't recall seeing this reported before.
Nore me and we've been using ZFS for years, but never filled a pool
with such known simultanious access + ZIL before
2. Is there anyway to stop the scrub?
Other than freeing up some space, I don't think so. If this is a test
pool that you don't need, you could try destroying it and re-creating
it - that may be quicker and easier than recovering the existing pool.
Artems trick of cat /dev/null > /tank2/<bigfile> worked and I've now
managed to stop the scrub :)
3. Surely low space should never prevent stopping a scrub?
As Artem noted, ZFS is a copy-on-write filesystem. It is supposed to
reserve some free space to allow metadata updates (stop scrubs, delete
files, etc) even when it is "full" but I have seen reports of this not
working correctly in the past. A truncate-in-place may work.
Yes it did thanks, but as you said if this metadata update was failing
due to out of space lends credability to the fact that the same lack of
space and hence failure to update metadata could have also caused the
corruption in the first place.
Its interesting to note that the zpool is reporting pleanty of free space
even when the root zfs volume was showing 0, so you would expect there
to be pleanty of space for it be able to stop the scrub but it appears
not which is definitely interesting and could point to the underlying
cause?
zpool list tank2
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank2 19T 18.7T 304G 98% 1.00x ONLINE -
zfs list tank2
NAME USED AVAIL REFER MOUNTPOINT
tank2 13.3T 0 13.3T /tank2
Current state is:-
scan: scrub in progress since Wed Oct 31 16:13:53 2012
1.64T scanned out of 18.7T at 62.8M/s, 79h12m to go
280M repaired, 8.76% done
Something else that was interesting is while the scrub was running
devd was using a good amount of CPU 40% of a 3.3Ghz core, which I've
never seen before. Any ideas why its usage would be so high?
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmas...@multiplay.co.uk.
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
