On Fri, Sep 14, 2012 at 10:17 AM, Paul Graydon <p...@paulgraydon.co.uk>wrote:
> On 09/14/2012 05:37 AM, Brent Chapman wrote: > >> Aggregated network links involving multiple parallel circuits generally >> use some sort of hash on the 5-tuple (src ip, dest ip, protocol, src port, >> dest port) so that packets for a given TCP or UDP session all get sent down >> the same parallel circuit; this is an easy way to help ensure that the >> packets don't get re-ordered, which many protocols are sensitive to. >> However, if the particular "same parallel circuit" that they get sent down >> is broken, as appears to have been the case here, you can wind up with >> behavior like what you saw: certain sessions (that happen to get hashed >> down those broken circuits) break horribly, while others (that get hashed >> down non-broken circuits) are just fine. >> >> >> I've never really delved into the networking aspects of aggregation, > it's never been something I've had any need to utilise, so forgive me if > these are stupid questions. > > Under circumstances with which a port goes down, would the link > aggregation generally be fine? The system would presumably be smart enough > identify that the port isn't working and stop routing traffic that way? Yes, the link aggregation mechanism would typically take the down port out of service, and adjust the hashing function to hash over (N - 1) links instead of N links. > I'm assuming the failure in this case is that the packet loss was too > slight enough to disrupt the aggregation, but disruptive enough to mess > things up? > Yes, that's what I would assume. Depending on your hardware and configuration, "too slight" might be anything short of 100% packet loss; even might not trigger an aggregation change if the link status stays "up". The general bias is towards keeping the link as part of the aggregate; it is debatable whether or not that's a _good_ bias, but that's the usual default. > Why would this disrupt TCPs guarantee processes (admittedly I'm assuming > the application traffic was TCP and not UDP)? Presumably the packet would > fail to reach the other side so the sender would resend having failed to > get an ack? > > Well, since Andrew originally identified this as a "TCP window 0" problem, I'm pretty sure it was using TCP... ;-) Every time you timeout and resend, you drop the window size; eventually, the window size reaches 0. I think Andrew indicated that they found some miscabling as well as some flaky links. The miscabling could have easily caused 100% packet loss, but only in one direction (imagine a fiber pair, where one fiber in the pair is properly connected, but the other isn't; packets would flow just fine in one direction, but not the other), for example. I can imagine a scenario where the first few packets of a TCP connection get hashed one way (down a working link), allowing the session to be established, but the rest of the packets (once the connection is established) get hashed a different way (down a dead or unidirectionally-dead link), which will quickly cause the window size to drop to 0. -Brent > Paul > > ______________________________**_________________ > Tech mailing list > Tech@lists.lopsa.org > https://lists.lopsa.org/cgi-**bin/mailman/listinfo/tech<https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech> > This list provided by the League of Professional System Administrators > http://lopsa.org/ > >
_______________________________________________ Tech mailing list Tech@lists.lopsa.org https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
