Re: TCP connection issues against Amazon S3

2015-01-08 Thread Rick Jones
Strange thing is that sender does not misbehave at the beginning when receiver window is still small. Only after a while. Just guessing, but when the receiver window is small, the sender cannot get a large quantity of data out there at once, so any string of lost packets will tend to be smal

Re: TCP connection issues against Amazon S3

2015-01-08 Thread Eric Dumazet
On Thu, 2015-01-08 at 17:47 +, Erik Grinaker wrote: > FWIW, I've done a bisection, and it’s triggered by this change: > > https://github.com/torvalds/linux/commit/4e4f1fc226816905c937f9b29dabe351075dfe0f This totally makes sense, thanks for doing the bisection ! > > > We are not going to

Re: TCP connection issues against Amazon S3

2015-01-08 Thread Erik Grinaker
On 07 Jan 2015, at 21:33, Eric Dumazet wrote: > On Wed, 2015-01-07 at 20:37 +, Erik Grinaker wrote: > I agree. I have contacted Amazon about this, but am not too hopeful >> for a quick fix; they have been promising SACK-support on their >> loadbalancers since 2006, for example. >> >> That sai

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Yuchung Cheng
On Wed, Jan 7, 2015 at 12:37 PM, Erik Grinaker wrote: > On 07 Jan 2015, at 15:58, Eric Dumazet wrote: >> On Wed, 2015-01-07 at 13:31 +, Erik Grinaker wrote: >>> On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > >> On 06 Jan 20

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Eric Dumazet
On Wed, 2015-01-07 at 20:37 +, Erik Grinaker wrote: > I agree. I have contacted Amazon about this, but am not too hopeful > for a quick fix; they have been promising SACK-support on their > loadbalancers since 2006, for example. > > That said, since this change breaks a service as popular as

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Erik Grinaker
On 07 Jan 2015, at 15:58, Eric Dumazet wrote: > On Wed, 2015-01-07 at 13:31 +, Erik Grinaker wrote: >> On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: >>> On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > On 06 Jan 2015, at 20:26, Erik Grinaker wrote: This still doesn’t e

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Eric Dumazet
On Wed, 2015-01-07 at 13:31 +, Erik Grinaker wrote: > On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: > > On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > >> > >>> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: > >> This still doesn’t explain why it works with older kernels, but not ne

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Erik Grinaker
On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: > On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: >> >>> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: >> This still doesn’t explain why it works with older kernels, but not newer >> ones. I’m thinking it’s > probably some minor change, whic

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Erik Grinaker
On 07 Jan 2015, at 01:23, Lukas Tribus wrote: >> This still doesn’t explain why it works with older kernels, but not newer >> ones. > > Can you try the different 3.12-rc kernels? The information that this was > introduced in 3.12-rc1 as opposed to a specific -rc>1 releases may help > the guys he

RE: TCP connection issues against Amazon S3

2015-01-06 Thread Lukas Tribus
> This still doesn’t explain why it works with older kernels, but not newer > ones. Can you try the different 3.12-rc kernels? The information that this was introduced in 3.12-rc1 as opposed to a specific -rc>1 releases may help the guys here to pinpoint what exactly caused the behavior change on

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Yuchung Cheng
On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > >> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: >> >>> >>> On 06 Jan 2015, at 20:13, Eric Dumazet wrote: >>> >>> On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: >>> The transfer on the functioning Netherlands server does indee

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: > >> >> On 06 Jan 2015, at 20:13, Eric Dumazet wrote: >> >> On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: >> >>> The transfer on the functioning Netherlands server does indeed use SACKs, >>> while the Norway servers do not. >>> >>>

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 20:13, Eric Dumazet wrote: > > On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: > >> The transfer on the functioning Netherlands server does indeed use SACKs, >> while the Norway servers do not. >> >> For what it’s worth, I have made stripped down pcaps for a singl

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: > The transfer on the functioning Netherlands server does indeed use SACKs, > while the Norway servers do not. > > For what it’s worth, I have made stripped down pcaps for a single failing > transfer as well as a single functioning transfe

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
On 06 Jan 2015, at 19:16, Rick Jones wrote: > >>> A packet dump [1] shows repeated ACK retransmits for some of the >> TCP does not retransmit ACK ... do you mean DUPACKs sent by the receiver? >> >> I am trying to understand the problem. Could you confirm that it's the >> HTTP responses sent

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Rick Jones
On 01/06/2015 11:16 AM, Rick Jones wrote: I'm assuming one incident starts at XX:41:24.748265 in the trace? That does look like it is slowly slogging its way through a bunch of lost traffic, which was I think part of the problem I was seeing with the middlebox I stepped in, but I don't think I s

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 19:18, Yuchung Cheng wrote: > > On Tue, Jan 6, 2015 at 11:01 AM, Erik Grinaker wrote: >> >>> On 06 Jan 2015, at 18:33, Yuchung Cheng wrote: >>> >>> On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: > On 06 Jan 2015, at 17:20, Eric Dumazet wrote: > On

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Yuchung Cheng
On Tue, Jan 6, 2015 at 11:01 AM, Erik Grinaker wrote: > >> On 06 Jan 2015, at 18:33, Yuchung Cheng wrote: >> >> On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: >>> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: >> On 06

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Rick Jones
A packet dump [1] shows repeated ACK retransmits for some of the TCP does not retransmit ACK ... do you mean DUPACKs sent by the receiver? I am trying to understand the problem. Could you confirm that it's the HTTP responses sent from Amazon S3 got stalled, or HTTP requests sent from the receive

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 18:33, Yuchung Cheng wrote: > > On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: >> >>> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: >>> On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: > On 06 Jan 2015, at 16:04, Eric Dumazet wrote: > On Tue, 2015-0

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Yuchung Cheng
On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: > >> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: >> On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: On 06 Jan 2015, at 16:04, Eric Dumazet wrote: On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: > (CCing Yuchung

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 18:17 +, Erik Grinaker wrote: > Yes, pcap was taken on receiver (195.159.221.106). > > > If the sender is broken, changing the kernel on receiver wont help. > > > > BTW not using sack (on 54.231.132.98) is terrible for performance in > > lossy environments. > > It may

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: > On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: >>> On 06 Jan 2015, at 16:04, Eric Dumazet wrote: >>> On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: (CCing Yuchung, as his name comes up in the relevant commits) Af

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: > > On 06 Jan 2015, at 16:04, Eric Dumazet wrote: > > On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: > >> (CCing Yuchung, as his name comes up in the relevant commits) > >> > >> After upgrading from Ubuntu 12.04.5 to 14.04.1 we have

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 16:04, Eric Dumazet wrote: > On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: >> (CCing Yuchung, as his name comes up in the relevant commits) >> >> After upgrading from Ubuntu 12.04.5 to 14.04.1 we have begun seeing >> intermittent TCP connection hangs for HTTP image

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: > (CCing Yuchung, as his name comes up in the relevant commits) > > After upgrading from Ubuntu 12.04.5 to 14.04.1 we have begun seeing > intermittent TCP connection hangs for HTTP image requests against > Amazon S3. 3-5% of requests will sud