I posted a thread on this once long ago[1] -- but we're still fighting
with this problem and I wanted to throw it out here again.
All of our hardware is from Silicon Mechanics (SuperMicro chassis and
motherboards).
Up until now, all of the hardware has had a single 24-disk expander /
backplane -- but we recently got one of the new SC847-based models with
24 disks up front and 12 in the back -- a dual backplane setup.
We're using two SSD's in the front backplane as mirrored ZIL/OS (I
don't think we have the 4K alignment set up correctly) and two drives
in the back as L2ARC.
The rest of the disks are 1TB SATA disks which make up a single large
zpool via three 8-disk RAIDZ2's. As you can see, we don't have the
server maxed out on drives...
In any case, this new server gets between 400 and 600 of these timeout
errors an hour:
Aug 21 03:10:17 dev-zfs1 scsi: [ID 365881 kern.info]
/p...@0,0/pci8086,3...@8/pci15d9,1...@0 (mpt0):
Aug 21 03:10:17 dev-zfs1 Log info 31126000 received for target 8.
Aug 21 03:10:17 dev-zfs1 scsi_status=0, ioc_status=804b, scsi_state=c
Aug 21 03:10:17 dev-zfs1 scsi: [ID 365881 kern.info]
/p...@0,0/pci8086,3...@8/pci15d9,1...@0 (mpt0):
Aug 21 03:10:17 dev-zfs1 Log info 31126000 received for target 8.
Aug 21 03:10:17 dev-zfs1 scsi_status=0, ioc_status=804b, scsi_state=c
Aug 21 03:10:17 dev-zfs1 scsi: [ID 107833 kern.warning] WARNING:
/p...@0,0/pci8086,3...@8/pci15d9,1...@0/s...@8,0 (sd0):
Aug 21 03:10:17 dev-zfs1 Error for Command: write(10)
Error Level: Retryable
Aug 21 03:10:17 dev-zfs1 scsi: [ID 107833 kern.notice] Requested Block:
21230708 Error Block: 21230708
Aug 21 03:10:17 dev-zfs1 scsi: [ID 107833 kern.notice] Vendor: ATA
Serial Number: CVEM002600EW
Aug 21 03:10:17 dev-zfs1 scsi: [ID 107833 kern.notice] Sense Key: Unit
Attention
Aug 21 03:10:17 dev-zfs1 scsi: [ID 107833 kern.notice] ASC: 0x29 (power on,
reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0
Aug 21 03:10:21 dev-zfs1 scsi: [ID 365881 kern.info]
/p...@0,0/pci8086,3...@8/pci15d9,1...@0 (mpt0):
iostat -xnMCez shows that the first of the two ZIL drives receives
about twice the number of "errors" as the second drive.
There are no other errors on any other drives -- including the L2ARC
SSD's and the ascv_t times seem reasonably low and don't indicate a bad
drive to my eyes...
The timeouts above exact a rather large performance penalty on the
system, both in IO and general usage from an SSH console. Obvious
pauses and glitches when accessing the filesystem.
The problem _follows_ the ZIL and isn't tied to hardware. IOW, if I
switch to using the L2ARC drives as ZIL, those drives suddenly exhibit
the timeout problems...
If we connect the SSD drives directly to the LSI controller instead of
hanging off the hot-swap backplane, the timeouts go away.
If we use SSD's attached to the SATA controllers as ZIL, there are also
no performance issues or timeout errors.
So the problem only occurs with SSD drives acting as ZIL attached to
the backplane.
This is leading me to believe we have a driver issue of some sort in
the mpt subsystem unable to cope with the longer command path of
multiple backplanes. Someone alluded to this in [1] as well, and it
makes sense to me.
One quick fix to me would seem to be upping the SCSI timeout values.
How do you do this with the mpt driver?
We haven't yet been able to try OpenSolaris or Nexenta on one of these
systems to see if the problem goes away in later releases of the kernel
or driver, but I'm curious if anyone out there has any bright ideas as
to what we might be running into here and what's involved in fixing it.
We've swapped out backplanes and drives and the problem happens on
every single Silicon Mechanics system we have, so at this point I'm
really doubting it's a hardware issue :)
Hardware details are as follows:
Silicon Mechanics Storform iServ R518
(Based on SuperMicro SC847E16-R1400 chassis)
SuperMicro X8DT3 motherboard w/ onboard LSI1068 controller.
- One LSI port goes to the front backplane (where the bulk of the
SATA drives are, the two SSD's used as ZIL/OS)
- The other LSI port goes to the rear backplane where the two L2ARC
drives are along with a couple SATA's)
We've got 6GB's of RAM and 2 quad core Xeons in the box as well.
The SSD's themselves are all Intel X-25E's (32GB) with firmware 8860
and the LSI 1068 is a SAS1068E B3 with firmware 011c0200 (1.28.02.00).
We're running Solaris 10U8 mostly up to date and MPT HBA Driver v1.92.
Thoughts, theories and conjectures would be much appreciated... Sun
these days wants us to be able to reproduce the problem on Sun hardware
to get much support... Silicon Mechanics has been helpful, but they
don't have a large enough inventory on hand to replicate our hardware
setup it seems. :(
Ray
[1] http://markmail.org/message/gfz2cui2iua4dxpy
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
