Thanks, Working on a similar design and now know what to avoid :)
Rgds dan On Feb 27, 2012, at 4:22 PM, David Swafford wrote: > Hi Everyone! > > I had several requests for more feedback on our FCoE experience, based on > my comments from a thread last week, so I'm writing here with a bit more > background on our project in hopes that it saves some pain for others :-). > > I'm with a sizable health insurance provider in the mid-west, and we've > typically focused on technology vs. headcount as an overal strategy. Based > on that, we upgrade much more often than some of our peers in the industry > because techology is still cheaper than long-term staffing costs. > > Last fall, we were faced with an issue of both power and rack capacity > constraints in our primary datacenter, which is just three years old now. > As various ideas were on the table, which included taking out a section of > IT cubes to expand the DC, the most appealing idea was to consolidate our > server and network infrastructure into what was coined our "High Density > Row". > > We transitioned from Cat6500s as access to a Nexus 5K deployment, using 5Ks > as both distribution and access for the new HD row. We didn't like how > oversubscription is handled on 2K FEXs when it comes to 10G links, so for > the situation here all 5Ks made the most sense. Our capacity needs > couldn't justify 7Ks and while they would have been cool to have, we didn't > want to blow money just because. > > Our SAN is an EMC Symmetrix with Cisco MDS switches in between it and the > hosts (Fiber Channel). In the new row, we deployed all hosts with CNAs > (converged net adapters), which combine both FCoE storage and network in a > single 10Gb connection. Since FCoE was new to all of us, we use a phased > approach that the Nexus offered where we brough straight fiber channel > connections into our distibution layer 5Ks and used the Nexus' FCoE proxy > functionality to convert between true FC to FCoE. > > From the host perpsective, it was only aware of FCoE connectivity to the > Nexus. VSANs had to be created on the Nexus to map back to the FC VSANs on > the MDS side, Virtual Fiber Channel (VFC) interfaces were created on the > Nexus side, and a few other settings had to be configured. > > Overall though, the config wasn't huge, but the biggest hurdle for was that > as the network guys, we had to learn the storage side to be able to > properly set this up. So new terms like WWN (world wide name), floggy > database, VSAN (a VLAN for storage), etc. Also, on the Nexus side, you > have to enable the feature of FCOE, as Nexus OS is very modulular and > leaves most options disabled during the initial setup. > > The painful part, which is probably what might be of most interest here, is > that we hit a very strange and catrastrophic issue specific to QLogic's > 8242 Copper-based (twinax) CNA adapter. As part of the burn-in testing, we > were working with our server team to simulate the loss of a > link/card/switch (all hosts were dual-connected with dual-CNAs to separate > 5Ks). We were using the Cisco branded twinax cabling and QLogic's 8242 > card (brand new HP DL580s in this case, new card, new 5K, new cabling). > When a single link was dropped/diconnected PHYSICALLY (a shut/no shut is > not the same here), the host's throughput on BOTH storage and network went > to crap. > > Our baseline was showing nearly 400MB/s on storage (raw disk IO) tests > prior to a link drop and 1-8 MB/s after! This siutation would not recover > until you fully rebooted/power cycled the server. We had the same results > accross every HP DL 580 tested, which was 5-6 of them I belive. We > replaced CNAs, cables, and even moved ports across 5Ks. It didn't matter > which cable, 5K, port, of card we used, all reacted the same! The hosts > were all Windows 2008 Datacenter, simliar hardware, Nexus 5K on current > code, twinax cabling. > > This situation led to a sev 2 w/ Cisco, the equivalant w/ HP, EMC, and > QLogic. We used both the straight QLogic 8242 and the HP OEM'd version and > the results were identical. QLogic acknowledged the issue but could not > resolve it due not being able to grab a hardware level trace of the > connection (required some type of test equipment that they couldn't provide > and we didn't have). > > As part of our trail/error testing, we had our re-seller ship us the fiber > versions of the same QLogic cards, becuase we eventually got down to a gut > instinct of this being a copper/electrical anomoly. That instict was > dead-on. Switching to the fiber versions, with fiber SFPs on the 5K side > resolved the situation entirely. We are now able to drop a link with NO > noticable degradation, back and forth, and eveyrthing is consistent again. > > We originally went the twinax route because it was signifiantly cheaper > than the fiber, but in retrospect, as a whole, the danger posed was not > worth it. You might ask, well... why would you intentially drop the > cable? Think about a situation of doing a code upgrade on the 5K, since > it's not a dual-sup box, you physcailly go through a reboot to upgrade it. > That reboot right htere would have hosed our entire environment (keep in > mind, the HD row's intent was to replace a signifiant portion of our > production environment). You could also have a HW failure on a 5K. It > kind of defeats the point of all this redundancy if your throuhput goes to > hell when loosing a single path. As our storage guys best put it "i'd > rather loose a path than have bad performance through it....based on how > things alert, I'd know right away if a path were down, but not if it were > severaly degraded." > > Btw, we've been rock solid on the fiber-connected CNAs ever since. We're > still using copper on our connections to HP blade chassis though, which go > to FLEX Fabric cards, as we couldn't produce the problem on those. For > those wondering, we did rebuild several of the DL580s from scratch (all of > this was a new deployment, thankfully!), we also went through many > iterations of driver updates/changes/etc. > > Lots of head-banging and teamwork eventually got us squared away! This > situation is a good example of why network guys NEED to have a great > relationship with both server and storage guys (we're all really close > where I'm at). Had there been tension/etc between the teams, this would > have been signifiantky harder to resolve. > > Hope this helps, sorry for the long winded email :-), but I think those > interested will find it beneficial. > > > David.
