Based on recommendations from this list, I asked the company that built
my box to use an LSI SAS3081E controller.
The first problem I noticed was that the drive-numbers were ordered
incorrectly. That is, given that my system has 24 bays (6 rows, 4
bays/row), the drive numbers from top-to-bottom & left-to-right were 6,
1, 0, 2, 4, 5 - even though when the system boots, each drive is scanned
in perfect order (I can tell by watching the LEDs blink).
I contacted LSI tech support and they explained:
<start response>
SAS treats device IDs differently than SCSI. LSI SAS controllers
"remember" devices in the order they were discovered by the controller.
This memory is persistent across power cycles. It is based on the world
wide name (WWN) given uniquely to every SAS device. This allows your
boot device to remain your boot device no matter where it migrates in
the SAS topology.
In order to clear the memory of existing devices you need at least one
device that will not be present in your final configuration. Re-boot
the machine and enter the LSI configuration utility (CTRL-C). Then find
your way to SAS Topology. To see "more" options, press CTRL-M. Choose
the option to clear all non-present device IDs. This clears the
persistent memory of all devices not present at that time. Exchange the
drives. The system will now remember the order it finds the drives
after the next boot cycle.
<end response>
Sure enough, I was able to physical reorder my drives so they were 0, 1,
2, 4, 5, 6 - so, appearantly, the company that put my system together
moved the drives around after they were initially scanned. But where is
3? (answer below). Then I tried another test:
1. make first disk blink
# run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10
10+0 records in
10+0 records out
2. pull disk '0' out and replace it with a brand new disk
# run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10
dd: /dev/dsk/c2t0d0p0: open: No such file or directory
3. scratch head and try again with '3' (I had previously cleared the
LSI's controllers memory)
# run dd if=/dev/dsk/c2t3d0p0 of=/dev/null count=10
10+0 records in
10+0 records out
So, it seems my SAS controller is being too smart for its own good - it
tracks the drives themselves, not the drive-bays. If I hot-swap a brand
new drive into a bay, Solaris will see it as a new disk, not a
replacement for the old disk. How can ZFS support this? I asked the
LSI tech support again and got:
<start quote>
I don't have the knowledge to answer that, so I'll just say
this: most vendors, including Sun, set up the SAS HBA to use
"enclosure/slot" naming, which means that if a drive is
swapped, it does NOT get a new name (after all, the enclosure
and slot did not change).
<end quote>
So, now I turn to you... Here some information about my system:
Specs:
Motherboard: SuperMicro H8DME-2 Rev 2.01
- BIOS: AMI v2.58
HBA: LSI SAS3081E (SN: P068170707) installed in Slot #5
- LSI Configuration Utility v6.16.00.00 (2007-05-07)
Backplane: CI-Design 12-6412-01BR
HBA connected to BP via two SFF-8087-SFF-8087 cables
OS: SXCE b74
Details:
* Chassis has 24 SAS/SATA bays
* There are 6 backplanes - one for each *row* of drives
* I currently have only 6 drives installed (see pic)
* The LSI card is plugged into backplanes 1 & 2
* The LSI card is NOT configured to do any RAID - its only
JBOD as I'm using Solaris's ZFS (software-RAID)
Question:
* I only plan to use SATA drives, would using a SATA controller like
Supermicro's
AOC-SAT2-MV8 help?
Thanks again,
Kent
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
