Re: TCP connection issues against Amazon S3

2015-01-08 Thread Rick Jones
Strange thing is that sender does not misbehave at the beginning when receiver window is still small. Only after a while. Just guessing, but when the receiver window is small, the sender cannot get a large quantity of data out there at once, so any string of lost packets will tend to be smal

Re: TCP connection issues against Amazon S3

2015-01-08 Thread Eric Dumazet
On Thu, 2015-01-08 at 17:47 +, Erik Grinaker wrote: > FWIW, I've done a bisection, and it’s triggered by this change: > > https://github.com/torvalds/linux/commit/4e4f1fc226816905c937f9b29dabe351075dfe0f This totally makes sense, thanks for doing the bisection ! > > > We are not going to

Re: TCP connection issues against Amazon S3

2015-01-08 Thread Erik Grinaker
On 07 Jan 2015, at 21:33, Eric Dumazet wrote: > On Wed, 2015-01-07 at 20:37 +, Erik Grinaker wrote: > I agree. I have contacted Amazon about this, but am not too hopeful >> for a quick fix; they have been promising SACK-support on their >> loadbalancers since 2006, for example. >> >> That sai

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Yuchung Cheng
On Wed, Jan 7, 2015 at 12:37 PM, Erik Grinaker wrote: > On 07 Jan 2015, at 15:58, Eric Dumazet wrote: >> On Wed, 2015-01-07 at 13:31 +, Erik Grinaker wrote: >>> On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > >> On 06 Jan 20

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Eric Dumazet
On Wed, 2015-01-07 at 20:37 +, Erik Grinaker wrote: > I agree. I have contacted Amazon about this, but am not too hopeful > for a quick fix; they have been promising SACK-support on their > loadbalancers since 2006, for example. > > That said, since this change breaks a service as popular as

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Erik Grinaker
On 07 Jan 2015, at 15:58, Eric Dumazet wrote: > On Wed, 2015-01-07 at 13:31 +, Erik Grinaker wrote: >> On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: >>> On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > On 06 Jan 2015, at 20:26, Erik Grinaker wrote: This still doesn’t e

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Eric Dumazet
On Wed, 2015-01-07 at 13:31 +, Erik Grinaker wrote: > On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: > > On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > >> > >>> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: > >> This still doesn’t explain why it works with older kernels, but not ne

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Erik Grinaker
On 06 Jan 2015, at 22:00, Yuchung Cheng wrote: > On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: >> >>> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: >> This still doesn’t explain why it works with older kernels, but not newer >> ones. I’m thinking it’s > probably some minor change, whic

Re: TCP connection issues against Amazon S3

2015-01-07 Thread Erik Grinaker
On 07 Jan 2015, at 01:23, Lukas Tribus wrote: >> This still doesn’t explain why it works with older kernels, but not newer >> ones. > > Can you try the different 3.12-rc kernels? The information that this was > introduced in 3.12-rc1 as opposed to a specific -rc>1 releases may help > the guys he

RE: TCP connection issues against Amazon S3

2015-01-06 Thread Lukas Tribus
> This still doesn’t explain why it works with older kernels, but not newer > ones. Can you try the different 3.12-rc kernels? The information that this was introduced in 3.12-rc1 as opposed to a specific -rc>1 releases may help the guys here to pinpoint what exactly caused the behavior change on

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Yuchung Cheng
On Tue, Jan 6, 2015 at 1:04 PM, Erik Grinaker wrote: > >> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: >> >>> >>> On 06 Jan 2015, at 20:13, Eric Dumazet wrote: >>> >>> On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: >>> The transfer on the functioning Netherlands server does indee

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 20:26, Erik Grinaker wrote: > >> >> On 06 Jan 2015, at 20:13, Eric Dumazet wrote: >> >> On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: >> >>> The transfer on the functioning Netherlands server does indeed use SACKs, >>> while the Norway servers do not. >>> >>>

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 20:13, Eric Dumazet wrote: > > On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: > >> The transfer on the functioning Netherlands server does indeed use SACKs, >> while the Norway servers do not. >> >> For what it’s worth, I have made stripped down pcaps for a singl

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 19:42 +, Erik Grinaker wrote: > The transfer on the functioning Netherlands server does indeed use SACKs, > while the Norway servers do not. > > For what it’s worth, I have made stripped down pcaps for a single failing > transfer as well as a single functioning transfe

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
On 06 Jan 2015, at 19:16, Rick Jones wrote: > >>> A packet dump [1] shows repeated ACK retransmits for some of the >> TCP does not retransmit ACK ... do you mean DUPACKs sent by the receiver? >> >> I am trying to understand the problem. Could you confirm that it's the >> HTTP responses sent

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Rick Jones
On 01/06/2015 11:16 AM, Rick Jones wrote: I'm assuming one incident starts at XX:41:24.748265 in the trace? That does look like it is slowly slogging its way through a bunch of lost traffic, which was I think part of the problem I was seeing with the middlebox I stepped in, but I don't think I s

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 19:18, Yuchung Cheng wrote: > > On Tue, Jan 6, 2015 at 11:01 AM, Erik Grinaker wrote: >> >>> On 06 Jan 2015, at 18:33, Yuchung Cheng wrote: >>> >>> On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: > On 06 Jan 2015, at 17:20, Eric Dumazet wrote: > On

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Yuchung Cheng
On Tue, Jan 6, 2015 at 11:01 AM, Erik Grinaker wrote: > >> On 06 Jan 2015, at 18:33, Yuchung Cheng wrote: >> >> On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: >>> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: >> On 06

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Rick Jones
A packet dump [1] shows repeated ACK retransmits for some of the TCP does not retransmit ACK ... do you mean DUPACKs sent by the receiver? I am trying to understand the problem. Could you confirm that it's the HTTP responses sent from Amazon S3 got stalled, or HTTP requests sent from the receive

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 18:33, Yuchung Cheng wrote: > > On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: >> >>> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: >>> On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: > On 06 Jan 2015, at 16:04, Eric Dumazet wrote: > On Tue, 2015-0

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Yuchung Cheng
On Tue, Jan 6, 2015 at 10:17 AM, Erik Grinaker wrote: > >> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: >> On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: On 06 Jan 2015, at 16:04, Eric Dumazet wrote: On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: > (CCing Yuchung

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 18:17 +, Erik Grinaker wrote: > Yes, pcap was taken on receiver (195.159.221.106). > > > If the sender is broken, changing the kernel on receiver wont help. > > > > BTW not using sack (on 54.231.132.98) is terrible for performance in > > lossy environments. > > It may

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 17:20, Eric Dumazet wrote: > On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: >>> On 06 Jan 2015, at 16:04, Eric Dumazet wrote: >>> On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: (CCing Yuchung, as his name comes up in the relevant commits) Af

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 16:11 +, Erik Grinaker wrote: > > On 06 Jan 2015, at 16:04, Eric Dumazet wrote: > > On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: > >> (CCing Yuchung, as his name comes up in the relevant commits) > >> > >> After upgrading from Ubuntu 12.04.5 to 14.04.1 we have

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
> On 06 Jan 2015, at 16:04, Eric Dumazet wrote: > On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: >> (CCing Yuchung, as his name comes up in the relevant commits) >> >> After upgrading from Ubuntu 12.04.5 to 14.04.1 we have begun seeing >> intermittent TCP connection hangs for HTTP image

Re: TCP connection issues against Amazon S3

2015-01-06 Thread Eric Dumazet
On Tue, 2015-01-06 at 15:14 +, Erik Grinaker wrote: > (CCing Yuchung, as his name comes up in the relevant commits) > > After upgrading from Ubuntu 12.04.5 to 14.04.1 we have begun seeing > intermittent TCP connection hangs for HTTP image requests against > Amazon S3. 3-5% of requests will sud

TCP connection issues against Amazon S3

2015-01-06 Thread Erik Grinaker
(CCing Yuchung, as his name comes up in the relevant commits) After upgrading from Ubuntu 12.04.5 to 14.04.1 we have begun seeing intermittent TCP connection hangs for HTTP image requests against Amazon S3. 3-5% of requests will suddenly stall in the middle of the transfer before timing out. We