I recently started having some really oddball things happening under stress. This coincided with the machine being updated to 11.2-STABLE (FreeBSD 11.2-STABLE #1 r342918:) from 11.1.
Specifically, I get "errors" like this: (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 bb 08 00 01 00 00 length 131072 SMID 269 Aborting command 0xfffffe0001179110 mps0: Sending reset from mpssas_send_abort for target ID 37 (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 bc 08 00 01 00 00 length 131072 SMID 924 terminated ioc 804b loginfo 31140000 scsi 0 state c xfer 0 (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 ba 08 00 01 00 00 length 131072 SMID 161 terminated ioc 804b loginfo 31140000 scsi 0 state c xfer 0 mps0: Unfreezing devq for target ID 37 (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 bc 08 00 01 00 00 (da12:mps0:0:37:0): CAM status: CCB request completed with an error (da12:mps0:0:37:0): Retrying command (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 bb 08 00 01 00 00 (da12:mps0:0:37:0): CAM status: Command timeout (da12:mps0:0:37:0): Retrying command (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 ba 08 00 01 00 00 (da12:mps0:0:37:0): CAM status: CCB request completed with an error (da12:mps0:0:37:0): Retrying command (da12:mps0:0:37:0): READ(10). CDB: 28 00 af 82 ba 08 00 01 00 00 (da12:mps0:0:37:0): CAM status: SCSI Status Error (da12:mps0:0:37:0): SCSI status: Check Condition (da12:mps0:0:37:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da12:mps0:0:37:0): Retrying command (per sense data) The "Unit Attention" implies the drive reset. It only occurs on certain drives under very heavy load (e.g. a scrub.) I've managed to provoke it on two different brands of disk across multiple firmware and capacities, however, which tends to point away from a drive firmware problem. A look at the pool data shows /no /errors (e.g. no checksum problems, etc) and a look at the disk itself (using smartctl) shows no problems either -- whatever is going on here the adapter is recovering from it without any data corruption or loss registered on *either end*! The configuration is an older SuperMicro Xeon board (X8DTL-IF) and shows: mps0: <Avago Technologies (LSI) SAS2008> port 0xc000-0xc0ff mem 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> There is also a SAS expander connected to that with all but the boot drives on it (the LSI card will not boot from the expander so the boot mirror is directly connected to the adapter.) Thinking this might be a firmware/driver compatibility related problem I flashed the card to 20.00.07.00, which is the latest available. That made the situation **MUCH** worse; now instead of getting unit attention issues I got *controller* resets (!!) which invariably some random device (and sometimes more than one) in one of the pools to get detached, as the controller didn't come back up fast enough for ZFS and it declares the device(s) in question "removed". Needless to say I immediately flashed the card back to 19.00.00.00! This configuration has been completely stable on 11.1 for upwards of a year, and only started misbehaving when I updated the OS to 11.2. I've pounded the living daylights out of this box for a very long time on a succession of FreeBSD OS builds and up to 11.1 have never seen anything like this; if I had a bad drive, it was clearly the drive. Looking at the commit logs for the mps driver it appears there isn't much here that *could* be involved, unless there's an interrupt issue with some of the MSI changes that is interacting with my specific motherboard line. Any ideas on running this down would be appreciated; it's not easy to trigger it on the 19.0 firmware but on 20. I can force a controller reset and detach within a few minutes by running scrubs so if there are things I can try (I have a sandbox machine with the same hardware in it that won't make me cry much if I blow it up) that would great. Thanks! -- Karl Denninger k...@denninger.net <mailto:k...@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/
smime.p7s
Description: S/MIME Cryptographic Signature
