On January 23, 2007 8:53:30 AM +1100 "James C. McPherson"
<[EMAIL PROTECTED]> wrote:
Hi Frank,
Frank Cusack wrote:
Would you please expand upon this, because I'm really interested
in what your thoughts are..... since I work on Sun's SAS driver :)
SAS is limited, by the Solaris driver, to 16 devices.
Correct.
Not even that,
it's limited to devices with SCSI id's 0-15, so if you have 16 drives
and they start at id 10, well you only get access to 6 of them.
Why would you start your numbering at 10?
Because you don't have a choice. It is up to the HBA and getting it
to do the right thing (ie, what you want) isn't always easy. IIRC,
the LSI Logic HBA(s) I had would automatically remember SASAddress to
SCSI ID mappings. So if you had attached 16 drives, removed one
and replaced it with a different one (even in a JBOD, ie it would
be attached to the same PHY), it would be id 16, because the first 16
scsi id's (0-15) were already accounted for. And then the new drive,
lets call it a replacement for a failed drive, would be unaccessible
under Solaris.
Why it would ever start at something other than 0, I'm not sure. I
also kind of remember that scsi.conf had some setting to map the HBA
to target 7 (which doesn't apply to SAS! yet the reference there was
specifically for LSI 1068. again IIRC). I think that I was seeing
that drives started at 8 because of this initialization, and that
removing it allowed the drives to start at 0 -- once I reset the HBA
BIOS to forget the mappings it had already made.
But SAS doesn't even really have scsi target id's. It has WWN-like
identifiers. I guess HBAs do some kind of mapping but it's not
reliable and can change, and inspecting or hardcoding device->id
mappings requires changing settings in the card's BIOS/OF.
SAS has WWNs because that is what the standard requires. SAS hba
implementors are free to map WWNs to relatively user-friendly
identifiers, which is what the LSI SAS1064/SAS1064/E chips do.
Also, the HBA may renumber devices. That can be a big problem.
Agreed. No argument there!
It would be better to use the SASAddress the way the fibre channel
drivers use the WWN. Drives could still be mapped to scsi id's, but
it should be done by the Solaris driver, not the HBA. And when
multipathing the names should change like with FC.
That too is my preference. We're currently working on multipathing
with SAS.
That is good to hear.
That's one thing. The other is unreliability with many devices
attached. I've talked to others that have had this problem as well.
I offered to send my controller(s) and JBOD to Sun for testing, through
the support channel (I had a bug open on this for awhile), but they
didn't want it. I think it came down to the classic "we don't
sell that hardware" problem. The onboard SAS controllers (x4100, v215
etc) work fine due to the limited topology. I wonder how you fix
(hardcode) the scsi id's with those. Because you're not doing it
with a PCI card.
With a physically limited topology numbering isn't an issue because
of the way that the ports are connected to the onboard devices. It's
external devices (requiring a plugin hba) where it's potentially a
problem. Of course, to fully exploit that situation you'd need to
have 64K addressable targets attached to a single controller, and
that hasn't happened yet. So we do have a window of opportunity :)
I believe SAS supports a maximum of 128 devices per controller, including
multipliers.
-frank
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
