On Tue, Jan 6, 2015 at 11:01 AM, Erik Grinaker <e...@bengler.no> wrote: > >> On 06 Jan 2015, at 18:33, Yuchung Cheng <ych...@google.com> wrote: >> >> On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker <e...@bengler.no> wrote: >>> >>>> On 06 Jan 2015, at 17:20, Eric Dumazet <eric.duma...@gmail.com> wrote: >>>> On Tue, 2015-01-06 at 16:11 +0000, Erik Grinaker wrote: >>>>>> On 06 Jan 2015, at 16:04, Eric Dumazet <eric.duma...@gmail.com> wrote: >>>>>> On Tue, 2015-01-06 at 15:14 +0000, Erik Grinaker wrote: >>>>>>> (CCing Yuchung, as his name comes up in the relevant commits) >>>>>>> >>>>>>> After upgrading from Ubuntu 12.04.5 to 14.04.1 we have begun seeing >>>>>>> intermittent TCP connection hangs for HTTP image requests against >>>>>>> Amazon S3. 3-5% of requests will suddenly stall in the middle of the >>>>>>> transfer before timing out. We see this problem across a range of >>>>>>> servers, in several data centres and networks, all located in Norway. >>>>>>> >>>>>>> A packet dump [1] shows repeated ACK retransmits for some of the >> TCP does not retransmit ACK ... do you mean DUPACKs sent by the receiver? > > Ah, sorry, they are indeed DUPACKs; I thought they were the same thing. > >> I am trying to understand the problem. Could you confirm that it's the >> HTTP responses sent from Amazon S3 got stalled, or HTTP requests sent >> from the receiver (your host)? > > Yes. We run HTTP GET requests against S3 for images (typically a few megs in > size). Once in a while, the response transfer stalls about halfway through, > until the client (Curl) times out. The packet dump shows loads of DUPACKs > early on, then TCP retransmissions until the connection is closed.
Without SACK, the sender uses NewReno fast recovery and recovers one packet per RTT. In contrast, SACK-based fast recovery can potentially recover all lost packets in one RTT. I still can't explain the problem seen on newer kernel. But that got to be some receiver related changes, not 0f7cc9a3c2bd89b15720dbf358e9b9e62af27126 b/c it's a sender side change. > >> btw I suspect some middleboxes are stripping SACKOK options from your >> SYNs (or Amazon SYN-ACKs) assuming Amazon supports SACK. > > That may be. I just tested this on a server in the Netherlands, and I can not > reproduce the problem there, while I can reproduce it from multiple locations > and ISPs in Norway. Would it be helpful to have a packet dump from the > functioning Netherlands server as well? > > >>>>>>> requests. Using Ubuntu mainline kernels, we found the problem to have >>>>>>> been introduced between 3.11.10 and 3.12.0, possibly in >>>>>>> 0f7cc9a3c2bd89b15720dbf358e9b9e62af27126. The problem is also present >>>>>>> in 3.18.1. Disabling tcp_window_scaling seems to solve it, but has >>>>>>> obvious drawbacks for transfer speeds. Other sysctls do not seem to >>>>>>> affect it. >>>>>>> >>>>>>> I am not sure if this is fundamentally a kernel bug or a network >>>>>>> issue, but we did not see this problem with older kernels. >>>>>>> >>>>>>> [1] http://abstrakt.bengler.no/tcp-issues-s3.pcap.bz2 >>>>>> >>>>>> >>>>>> CC netdev >>>>>> >>>>>> This looks like the bug we fixed here : >>>>>> >>>>>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=39bb5e62867de82b269b07df900165029b928359 >>>>> >>>>> Has that patch gone into a release? Because the problem persists with >>>>> 3.18.1. >>>> >>>> Patch is in 3.18.1 yes. >>>> >>>> So thats a separate issue. >>>> >>>> Can you confirm pcap was taken at receiver (195.159.221.106), not sender >>>> (54.231.136.74) , and on which host is running the 'buggy kernel' ? >>> >>> Yes, pcap was taken on receiver (195.159.221.106). >>> >>>> If the sender is broken, changing the kernel on receiver wont help. >>>> >>>> BTW not using sack (on 54.231.132.98) is terrible for performance in >>>> lossy environments. >>> >>> It may well be that the sender is broken; however, the sender is Amazon S3, >>> so I do not have any control over it. And in any case, the problem goes >>> away with 3.11.10 on receiver, but persists with 3.12.0 (or later) on >>> receiver, so there must be some change in 3.12.0 which has caused this to >>> trigger. >>> >>> If you are confident that the problem is with Amazon, I can get in touch >>> with their engineering department. > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
