Re: [zfs-discuss] Resilver restarting several times

Jim Klimov Sat, 12 May 2012 04:56:52 -0700

2012-05-11 14:22, Jim Klimov wrote:

What conditions can cause the reset of the resilvering
process? My lost-and-found disk can't get back into the
pool because of resilvers restarting...


FOLLOW-UP AND NEW QUESTIONS

Here is a new piece of evidence - I've finally got something
out of fmdump - series of several (5) retries ending with a
fail, dated 75 seconds before resilvers restart (more below).
Not a squeak in zpool status nor dmesg nor /dev/console.

Guess I must assume that the disk is dying indeed, losing
connection or something like that after a random time (my
resilvers restart after 15min-5hrs), and at least a run of
SMART long diags is in order, while the pool would try to
rebuild onto another disk (the hotspare) instead of trying
to update this one which was in the pool.

Anyhow, information on the ex-pool disk is likely unused
anyway - from iostat I see that it is only written to,
with few to zero reads for minutes - so I'd lose nothing
by replacing it with a blank drive (that's strange though)...

I also guess that the disk gets found after something like
an unlogged bus reset or whatever, and this event causes
the resilvering to restart from scratch.
Q: Would this be the same in OI_151a, or would it continue
resilvering from where it left off? I think I had the pool
exported once during a resilver, and it restarted from the
same percentage counter, so it is possible ;)

Best course of action would be to get those people to fully
replace the untrustworthy disk... Or at least pull and push
it a bit - maybe it's contacts just got plain dirty/oxidized
and the disk should be re-seated in the enclosure...

I'd like someone to please confirm or deny my hypotheses
and guesses :)

DETAILS

According to format, the disk in tailed fmdump reports below
is indeed the one I'm trying to resilver into:

# format | gegrep -B1 '/pci@0,0/pci1022,7458@2/pci11ab,11ab@1/disk@2,0'
      10. c1t2d0 <ATA-SEAGATE ST32500N-3AZQ-232.88GB>
          /pci@0,0/pci1022,7458@2/pci11ab,11ab@1/disk@2,0

From the pool history we see the resilvering restart
timestamps... they closely match the retry-fail cycles,
following them by some 75 seconds:

# zpool history -il pond | tail; date
....

2012-05-12.10:43:35 [internal pool scrub done txg:91072311] complete=0[user root on thumper]2012-05-12.10:43:36 [internal pool scrub txg:91072311] func=1 mintxg=41maxtxg=91051854 [user root on thumper]2012-05-12.14:12:44 [internal pool scrub done txg:91072723] complete=0[user root on thumper]2012-05-12.14:12:45 [internal pool scrub txg:91072723] func=1 mintxg=41maxtxg=91051854 [user root on thumper]

Sat May 12 15:45:50 MSK 2012

And last but not least - the FMDUMP messages...

# fmdump -eV | tail -150

...
May 12 2012 10:42:19.559305872 ereport.io.scsi.cmd.disk.tran
nvlist version: 0
        class = ereport.io.scsi.cmd.disk.tran
        ena = 0x928e32c9d1700401
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = dev