Hi Mark, I'm going to throw out a guess here. By any chance, is the first octet of your router's MAC address a 4 or a 6? In general, modern routers do not load balance per-packet, which is what caused out-of-order issues in days gone. Load balancing is usually done based on a hash of the source and/or destination IP of the packet, MPLS label, or Ethernet (on a switched interface). The most common cause for actual unordering of packets/frames in a modern service provider network, in my experience, is actually this hashing mechanism. Many vendor's hashing implementations assume, based on position in the frame, that a frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame, not an MPLS frame. This can result in out of order packets. The most common fix is control word being applied on a pseudowire (assuming you are being carried across a pseudowire in the SP network), but if this *is* what is occurring, you could also resolve the issue by changing your MAC address.
_______________________ *Jason R. Rokeach* m: 603.969.5549 e: ja...@rokeach.net On Tue, Jan 17, 2017 at 2:12 PM, Mark Wicker <mwic...@esri.com> wrote: > Hi, > > I have 1G Level3 ethernet dedicated internet service as one of my ISP's at > my company based in the Los Angeles (Inland Empire) area. After seeing > strange application behavior while using this circuit, I failed it out of > service and have been troubleshooting it with a directly connected machine > (publically addressed, no firewall, nothing between this machine and our > Level3 router). I have taken several packet captures while accessing > various sites and have noticed large numbers of out of order packets which > are wreaking havoc with TCP connections and other traffic. In my > experience, per-packet load balancing across various different links can > cause this issue. I do not see this behavior with my other ISP's. I have > had several tickets opened with Level3 but have had no success. Any help > here? Anyone out there seen this and have any contacts that may be able to > help? > > > FYI - we own our own public IP space and advertise via BGP to Level3. > Currently I am using a dedicated /24 of our space advertised to Level3 only > to ensure that the return path is through Level3 and not another ISP. Also, > everything is single linked from a layer 2 and 3 perspective from the > router to the test machine to ensure that the cause of any out of order > packets is not on our end. > > > Thanks, > > > -- > Mark Wicker | Senior Network Engineer > Esri | 380 New York St | Redlands, CA 92354 | USA > T 909 793 2853 x2741 | mwic...@esri.com<mailto:mwic...@esri.com> | > esri.com<http://www.esri.com/> > >
