Re: LSI 9240-4i 4K alignment
On 16 Aug, Steven Hartland wrote: > - Original Message - > From: "George Kontostanos" > >> You are right, the chip specs say: LSISAS2108 RAID-on-Chip >> >> The drives are identified as mfisyspd0, mfisyspd1, etc. > > The following might be interesting to you:- > http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models > > Which states:- > LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical, > no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed > to LSI9211 IT/IR > > This is insteresting as this is the card we're using but > in the 8 port version under mps :) I wish I would have known this earlier. I just put together a ZFS server using LSI MegaRAID SAS 9240-8i cards. The cabling probably would have been cleaner with the 9211-8i, but I went with the 9240 because the vendor that I purchased the cards from listed that 9240 as being PCI-Express 2.0, but didn't say that about the 9211. I also got the impression that the 9240 recognized JBOD drives with the off-the-shelf firmware, whereas the 9211 did not. Even LSI's own site is a bit confusing. They list the 9211 in the HBA section, but its specs don't mention JBOD, whereas the 9240 is listed in the RAID section and its specs do list JBOD. If the only physical difference between the cards is the connector position, it seems odd that they don't offer products with all the combinations of firmware and connector position. I haven't configured the ZFS pool yet, but I didn't have any trouble installing FreeBSD 9.1-BETA on the GPT partitioned boot drive, which shows up as an mfi device. I'm planning on getting the ZFS pool up and running in the next few days. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: LSI 9240-4i 4K alignment
On 8 Aug, George Kontostanos wrote: > Hi all, > > We have a server with a LSI 9240-4i controller configured in JBOD with > 4 SATA disks. Running FreeBSD 9.1-Beta1: > > Relevant dmesg: > > FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012 > r...@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 > > Features=0xbfebfbff > Features2=0x1fbae3ff > AMD Features=0x28100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 17179869184 (16384 MB) > avail memory = 16471670784 (15708 MB) > ... > mfi0: port 0xe000-0xe0ff mem > 0xf7a6-0xf7a63fff,0xf7a0-0xf7a3 irq 16 at device 0.0 on > pci1 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > ... > mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host > mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started > (PCI ID 0073/1000/9241/1000) > mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 > mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started > (PCI ID 0073/1000/9241/1000) > mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 > mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107 > mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A > mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) > ... > mfisyspd0 on mfi0 > mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd0: SYSPD volume attached > mfisyspd1 on mfi0 > mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd1: SYSPD volume attached > mfisyspd2 on mfi0 > mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd2: SYSPD volume attached > mfisyspd3 on mfi0 > mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume > mfisyspd3: SYSPD volume attached > ... > mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info: > enclPd=, scsiType=0, portMap=00, > sasAddr=44332211, > mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) > mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info: > enclPd=, scsiType=0, portMap=02, > sasAddr=443322110200, > mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) > mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info: > enclPd=, scsiType=0, portMap=03, > sasAddr=443322110100, > mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) > mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info: > enclPd=, scsiType=0, portMap=01, > sasAddr=443322110300, > mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12 > 16:32:56; (28 seconds since power on) > > The problem: > > When trying to create a RaidZ pool using gpart and perform a 4K > alignment using gnop, we get the follwoing error immediately after > exporting the pool and destroying the .nop devices: > > id: 8043746387654554958 > state: FAULTED > status: One or more devices contains corrupted data. > action: The pool cannot be imported due to damaged devices or data. > The pool may be active on another system, but can be imported using > the '-f' flag. >see: http://illumos.org/msg/ZFS-8000-5E > config: > > Pool FAULTED corrupted data > raidz1-0ONLINE > 13283347160590042564 UNAVAIL corrupted data > 16981727992215676534 UNAVAIL corrupted data > 6607570030658834339 UNAVAIL corrupted data > 3435463242860701988 UNAVAIL corrupted data I'm planning on doing something similar, but I'm curious about how gnop and GPT labels interact. I want to partition the drives for my pool slightly on the small side so that I'm less likely to run into problems if I have to replace a drive in the future. If I used gpart to create and label a GPT partition on the drive, the partition will show up as /dev/gpt/label. The gnop man page says that running gnop on dev creates /dev/dev.nop. What happens if you gnop /dev/gpt/label? Is this what you are doing? > When we use glabel for the same purpose with the combination of gnop, > the pool imports fine. > > Any suggestions? It should be sufficient to only gnop one of the devices. You should be able to create the pool with only one gnop device to get the ashift value that you desire, export the pool, destroy the .nop device, and import the pool. If it things the device is corrupted (which seems like a bug of some sort), then ZFS should be able to resilver it. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: LSI 9240-4i 4K alignment
On Sun, Aug 19, 2012 at 8:58 PM, Don Lewis wrote: > On 8 Aug, George Kontostanos wrote: >> Hi all, >> >> We have a server with a LSI 9240-4i controller configured in JBOD with >> 4 SATA disks. Running FreeBSD 9.1-Beta1: >> >> Relevant dmesg: >> >> FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012 >> r...@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >> CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 >> >> Features=0xbfebfbff >> Features2=0x1fbae3ff >> AMD Features=0x28100800 >> AMD Features2=0x1 >> TSC: P-state invariant, performance statistics >> real memory = 17179869184 (16384 MB) >> avail memory = 16471670784 (15708 MB) >> ... >> mfi0: port 0xe000-0xe0ff mem >> 0xf7a6-0xf7a63fff,0xf7a0-0xf7a3 irq 16 at device 0.0 on >> pci1 >> mfi0: Using MSI >> mfi0: Megaraid SAS driver Ver 4.23 >> ... >> mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host >> mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started >> (PCI ID 0073/1000/9241/1000) >> mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 >> mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started >> (PCI ID 0073/1000/9241/1000) >> mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 >> mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107 >> mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A >> mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) >> ... >> mfisyspd0 on mfi0 >> mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd0: SYSPD volume attached >> mfisyspd1 on mfi0 >> mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd1: SYSPD volume attached >> mfisyspd2 on mfi0 >> mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd2: SYSPD volume attached >> mfisyspd3 on mfi0 >> mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd3: SYSPD volume attached >> ... >> mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info: >> enclPd=, scsiType=0, portMap=00, >> sasAddr=44332211, >> mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) >> mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info: >> enclPd=, scsiType=0, portMap=02, >> sasAddr=443322110200, >> mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) >> mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info: >> enclPd=, scsiType=0, portMap=03, >> sasAddr=443322110100, >> mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) >> mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info: >> enclPd=, scsiType=0, portMap=01, >> sasAddr=443322110300, >> mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12 >> 16:32:56; (28 seconds since power on) >> >> The problem: >> >> When trying to create a RaidZ pool using gpart and perform a 4K >> alignment using gnop, we get the follwoing error immediately after >> exporting the pool and destroying the .nop devices: >> >> id: 8043746387654554958 >> state: FAULTED >> status: One or more devices contains corrupted data. >> action: The pool cannot be imported due to damaged devices or data. >> The pool may be active on another system, but can be imported using >> the '-f' flag. >>see: http://illumos.org/msg/ZFS-8000-5E >> config: >> >> Pool FAULTED corrupted data >> raidz1-0ONLINE >> 13283347160590042564 UNAVAIL corrupted data >> 16981727992215676534 UNAVAIL corrupted data >> 6607570030658834339 UNAVAIL corrupted data >> 3435463242860701988 UNAVAIL corrupted data > > I'm planning on doing something similar, but I'm curious about how gnop > and GPT labels interact. I want to partition the drives for my pool > slightly on the small side so that I'm less likely to run into problems > if I have to replace a drive in the future. If I used gpart to create > and label a GPT partition on the drive, the partition will show up as > /dev/gpt/label. The gnop man page says that running gnop on dev creates > /dev/dev.nop. What happens if you gnop /dev/gpt/label? Is this what > you are doing? > >> When we use glabel for the same purpose with the combination of gnop, >> the pool imports fine. >> >> Any suggestions? > > It should be sufficient to only gnop one of the devices. You should be > able to create the pool with only one gnop device to get the ashift > value that you desire, export the pool, destroy the .nop device, and > import the pool. If it things the device is corrupted (which seems like > a bug of some sort), then ZFS should be able to resilver it. > After a few tests with many scenarios, the pool shows as ASHIFT=12 with the devices gnoped directly. However, the pe
Re: LSI 9240-4i 4K alignment
- Original Message - From: "Don Lewis" To: Cc: ; ; Sent: Sunday, August 19, 2012 6:37 PM Subject: Re: LSI 9240-4i 4K alignment On 16 Aug, Steven Hartland wrote: - Original Message - From: "George Kontostanos" You are right, the chip specs say: LSISAS2108 RAID-on-Chip The drives are identified as mfisyspd0, mfisyspd1, etc. The following might be interesting to you:- http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models Which states:- LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical, no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed to LSI9211 IT/IR This is insteresting as this is the card we're using but in the 8 port version under mps :) I wish I would have known this earlier. I just put together a ZFS server using LSI MegaRAID SAS 9240-8i cards. The cabling probably would have been cleaner with the 9211-8i, but I went with the 9240 because the vendor that I purchased the cards from listed that 9240 as being PCI-Express 2.0, but didn't say that about the 9211. I also got the impression that the 9240 recognized JBOD drives with the off-the-shelf firmware, whereas the 9211 did not. Even LSI's own site is a bit confusing. They list the 9211 in the HBA section, but its specs don't mention JBOD, whereas the 9240 is listed in the RAID section and its specs do list JBOD. If the only physical difference between the cards is the connector position, it seems odd that they don't offer products with all the combinations of firmware and connector position. I haven't configured the ZFS pool yet, but I didn't have any trouble installing FreeBSD 9.1-BETA on the GPT partitioned boot drive, which shows up as an mfi device. I'm planning on getting the ZFS pool up and running in the next few days. HBA's are the way to go if your using ZFS to manage the disks, you only need RAID if your using a FS which doesn't manage the disk side well such as UFS. Its often quite common for RAID controllers to actually be slower vs RAID controllers as the RAID stack can get in the way. JBOD is generally what HBA's do by default which may be the reason why LSI's site doesn't mention it. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: LSI 9240-4i 4K alignment
On 08/19/2012 14:04, Steven Hartland wrote: > > - Original Message - From: "Don Lewis" > To: > Cc: ; ; > > Sent: Sunday, August 19, 2012 6:37 PM > Subject: Re: LSI 9240-4i 4K alignment > > >> On 16 Aug, Steven Hartland wrote: >>> - Original Message - From: "George Kontostanos" >>> >>> You are right, the chip specs say: LSISAS2108 RAID-on-Chip The drives are identified as mfisyspd0, mfisyspd1, etc. >>> >>> The following might be interesting to you:- >>> http://forums.servethehome.com/showthread.php?599-LSI-RAID-Controller-and-HBA-Complete-Listing-Plus-OEM-Models >>> >>> >>> Which states:- >>> LSI MegaRAID SAS 9240-4i 1x4 port internal SAS vertical, >>> no cache, no BBU, RAID 0, 1, 10 and 5, can be crossflashed >>> to LSI9211 IT/IR >>> >>> This is insteresting as this is the card we're using but >>> in the 8 port version under mps :) >> >> I wish I would have known this earlier. I just put together a ZFS >> server using LSI MegaRAID SAS 9240-8i cards. The cabling probably would >> have been cleaner with the 9211-8i, but I went with the 9240 because the >> vendor that I purchased the cards from listed that 9240 as being >> PCI-Express 2.0, but didn't say that about the 9211. I also got the >> impression that the 9240 recognized JBOD drives with the off-the-shelf >> firmware, whereas the 9211 did not. >> >> Even LSI's own site is a bit confusing. They list the 9211 in the HBA >> section, but its specs don't mention JBOD, whereas the 9240 is listed in >> the RAID section and its specs do list JBOD. If the only physical >> difference between the cards is the connector position, it seems odd >> that they don't offer products with all the combinations of firmware and >> connector position. >> >> I haven't configured the ZFS pool yet, but I didn't have any trouble >> installing FreeBSD 9.1-BETA on the GPT partitioned boot drive, which >> shows up as an mfi device. I'm planning on getting the ZFS pool up and >> running in the next few days. > > HBA's are the way to go if your using ZFS to manage the disks, you only > need RAID if your using a FS which doesn't manage the disk side well > such as UFS. > > Its often quite common for RAID controllers to actually be slower > vs RAID controllers as the RAID stack can get in the way. > > JBOD is generally what HBA's do by default which may be the reason > why LSI's site doesn't mention it. > >Regards >Steve Just to clear up, The 9240 is a sas2008 based card with the megaraid software on top of it. In it's default config from LSI the FreeBSD mfi will recognize it in later versions of FreeBSD (The upcoming 9.1 for sure) Older versions of mfi will not recognize it. The card can be flashed with IT firmware and then becomes a 9211 HBA, but it's a bit more expensive than a 9211 is so that doesn't make sense to do in many cases. On the dmesg posted the firmware on the card is phase 11. This *must* be in lockstep with the driver version or the card may not play nicely. FreeBSD 8.3 and 9.0 have v13 of the driver, the upcoming 9.1 will have v14. Note that v14 fixes a *ton* of stability bugs, including issues where bad drives would hang the controller or prevent systems from booting. Thanks, Josh Paetzel ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
