On 28/02/11 02:08 AM, Dave Pooser wrote:
On 2/27/11 5:15 AM, "James C. McPherson"<j...@opensolaris.org> wrote:
On 27/02/11 05:24 PM, Dave Pooser wrote:
On 2/26/11 7:43 PM, "Bill Sommerfeld"<sommerf...@hamachi.org> wrote:
On your system, c12 is the mpxio virtual controller; any disk which is
potentially multipath-able (and that includes the SAS drives) will
appear as a child of the virtual controller (rather than appear as the
child of two or more different physical controllers).
Hmm... That makes sense, except that my drives are all SATA because I'm
cheap^H^H^H fiscally conservative. :^)
They're attached to a SAS hba, which is doing translations for them
using SATL - SAS to ATA Translation Layer.
Yeah, but they're still not multipathable, are they?
Not _physically_ multipathable, but the driver and the chip combine
their efforts to make this opaque to you. That way you, the user, only
have to worry about one path.
'stmsboot -L' displayed no mappings,
this is because mpt_sas(7d) controllers - which you have - are using
MPxIO by default. Running stmsboot -L will only show mappings if you've
enabled or disabled MPxIO....
but I went ahead and tried stmsboot
-d to disable multipathing;
... and now you have disabled MPxIO, stmsboot -L should show mappings.
Nope:
locadmin@bigdawg2:~# stmsboot -L
stmsboot: MPXIO disabled
Sorry, forgot one step (it's been a while since I actually disabled
MPxIO): you have to re-enable MPxIO :-)
after reboot instead of seeing nine disks on a
single controller I now see ten different controllers (in a machine that
has four PCI controllers and one motherboard controller):
This is a side effect of how your expanders are configured to operate
on your motherboard.
But there shouldn't be any expanders in the system-- the front backplane
has six SFF-8087 ports to control 24 drives, and the rear backplane has
three more SFF-8087 ports to control 12 more drives. Each of those ports
is connected directly to an SFF-8087 port on an LSI 9211-8i controller,
except that the ninth port is connected to the integrated LSI 2008
controller on the motherboard.
I misread your initial email, sorry.
So your disks are connected to separate PHYs on the HBA, by virtue
of their cabling. You can see this for yourself by looking at the
iport@xx element in the physical paths:
1. c13t5000CCA222DF92A0d0
/pci@0,0/pci8086,340a@3/pci1000,72@0/iport@10/disk@w5000cca222df92a0,0
2. c14t5000CCA222DF8FBEd0
/pci@0,0/pci8086,340e@7/pci1000,3020@0/iport@1/disk@w5000cca222df8fbe,0
The "xx" part is a bitmask, starting from 0, which gives you an
indication of which PHY the device is attached to.
Your disk #1 above is connected to iport@10, which is PHY #4 when
you have x1 ports:
PHY iport@
0 1
1 2
2 4
3 8
4 10
5 20
6 40
7 80
otoh, if you have x4 "wide" ports:
PHY iport@0
0 f (these are really 0f, but we drop the leading 0)
1 f
2 f
3 f
4 f0
5 f0
6 f0
7 f0
If you're lucky, your expanders and the enclosure that they're
configured into will show up with one or more SES targets. If
that's the case, you might be able to see bay numbers with the
fmtopo command - when you run it as root:
# /usr/lib/fm/fmd/fmtopo -V
If this doesn't work for you, then you'll have to resort to the
tried and tested use of dd to /dev/null for each disk, and see
which lights blink.
I can live with that-- but I really want to know what (real, not virtual)
controllers disks are connected to; I want to build 3 8-disk RAIDz2 vdevs
now (with room for a fourth for expansion later) and I really want to make
sure each of those vdevs has fewer than three disks per controller so a
single controller failure can degrade my vdevs but not kill them.
With the information above about the PHY/iport relationship, I
hope you can now see better what your physical layout is. Also,
please remember that using MPxIO means you have a single virtual
controller, and the driver stack handles the translation to physical
for you so you don't have to worry about that aspect. Of course,
if you want to worry about it, feel free.
Probably my next step is going to be to take a look with Nexenta Core or
FreeBSD (or maybe SolEx11 for a temporary eval) and see if either of those
gives me a saner view, but other suggestions would be appreciated.
Not sure where FreeBSD is up to, nor Nexenta Core. If Nexenta Core
includes changes from snv_118 onwards, then you'll have pretty much
the same mpt_sas(7d) driver that S11x has.
James C. McPherson
--
Oracle
http://www.jmcp.homeunix.com/blog
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
