Re: Major network performance regression in 3.7

2013-01-06 Thread John Stoffel
> "Willy" == Willy Tarreau writes: Willy> On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: >> On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: >> > > >> > > (sd->len is usually 4096, which is expected, but sd->total_len value is >> > > huge in your case, so we always set t

Re: Major network performance regression in 3.7

2013-01-06 Thread John Stoffel
> "Willy" == Willy Tarreau writes: Willy> On Sun, Jan 06, 2013 at 04:49:35PM -0500, John Stoffel wrote: >> > "Willy" == Willy Tarreau writes: >> Willy> On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: >> >> On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: >> >> > > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 04:49:35PM -0500, John Stoffel wrote: > > "Willy" == Willy Tarreau writes: > > Willy> On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: > >> On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: > >> > > > >> > > (sd->len is usually 4096, which is expecte

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:39:31AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote: > > > OK it works like a charm here now ! I can't break it anymore, so it > > looks like you finally got it ! > > > > I noticed that the data rate was higher when the loopback's

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 20:34 +0100, Willy Tarreau wrote: > OK it works like a charm here now ! I can't break it anymore, so it > looks like you finally got it ! > > I noticed that the data rate was higher when the loopback's MTU > is exactly a multiple of 4096 (making the 64k choice optimal) > whi

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:00:15AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: > > > > > > (sd->len is usually 4096, which is expected, but sd->total_len value is > > > huge in your case, so we always set the flag in fs/splice.c) > > > > I am testing : > >

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:51 -0800, Eric Dumazet wrote: > > > > (sd->len is usually 4096, which is expected, but sd->total_len value is > > huge in your case, so we always set the flag in fs/splice.c) > > I am testing : > >if (sd->len < sd->total_len && pipe->nrbufs > 1) >

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
> > (sd->len is usually 4096, which is expected, but sd->total_len value is > huge in your case, so we always set the flag in fs/splice.c) I am testing : if (sd->len < sd->total_len && pipe->nrbufs > 1) more |= MSG_SENDPAGE_NOTLAST; -- To unsubscribe from this list: se

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:39 -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 18:35 +0100, Willy Tarreau wrote: > > > Unfortunately it does not work any better, which means to me > > that we don't leave via this code path. I tried other tricks > > which failed too. I need to understand this part

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 18:35 +0100, Willy Tarreau wrote: > Unfortunately it does not work any better, which means to me > that we don't leave via this code path. I tried other tricks > which failed too. I need to understand this part better before > randomly fiddling with it. > OK, now I have you

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 09:10:55AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 17:44 +0100, Willy Tarreau wrote: > > On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: > > > Hmm, I'll have to check if this really can be reverted without hurting > > > vmsplice() again. > > > > Loo

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 17:44 +0100, Willy Tarreau wrote: > On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: > > Hmm, I'll have to check if this really can be reverted without hurting > > vmsplice() again. > > Looking at the code I've been wondering whether we shouldn't transform > the

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 16:51 +0100, Willy Tarreau wrote: > Hi Eric, > > Oh sorry, I didn't really want to pollute the list with links and configs, > especially during the initial report with various combined issues :-( > > The client is my old "inject" tool, available here : > > http://git.

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 08:39:53AM -0800, Eric Dumazet wrote: > Hmm, I'll have to check if this really can be reverted without hurting > vmsplice() again. Looking at the code I've been wondering whether we shouldn't transform the condition to perform the push if we can't push more segments, but I

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
Hi Eric, On Sun, Jan 06, 2013 at 06:59:02AM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 10:24 +0100, Willy Tarreau wrote: > > > It does not change anything to the tests above unfortunately. It did not > > even stabilize the unstable runs. > > > > I'll check if I can spot the original comm

Re: Major network performance regression in 3.7

2013-01-06 Thread Eric Dumazet
On Sun, 2013-01-06 at 10:24 +0100, Willy Tarreau wrote: > It does not change anything to the tests above unfortunately. It did not > even stabilize the unstable runs. > > I'll check if I can spot the original commit which caused the regression > for MTUs that are not n*4096+52. Since you don't p

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 11:25:25AM +0100, Willy Tarreau wrote: > OK good news here, the performance drop on the myri was caused by a > problem between the keyboard and the chair. After the reboot series, > I forgot to reload the firmware so the driver used the less efficient > firmware from the NIC

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 12:46:58PM +0100, Romain Francoise wrote: > Willy Tarreau writes: > > > That makes me think that I should try 3.8-rc2 since LRO was removed > > there :-/ > > Better yet, find a way to automate these tests so they can run continually > against net-next and find problems ea

Re: Major network performance regression in 3.7

2013-01-06 Thread Romain Francoise
Willy Tarreau writes: > That makes me think that I should try 3.8-rc2 since LRO was removed > there :-/ Better yet, find a way to automate these tests so they can run continually against net-next and find problems early... -- To unsubscribe from this list: send the line "unsubscribe linux-kernel

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sun, Jan 06, 2013 at 10:24:35AM +0100, Willy Tarreau wrote: > But before that I'll try to find the recent one causing the myri10ge to > slow down, it should take less time to bisect. OK good news here, the performance drop on the myri was caused by a problem between the keyboard and the chair.

Re: Major network performance regression in 3.7

2013-01-06 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 11:35:24PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 03:52 +0100, Willy Tarreau wrote: > > > OK so I observed no change with this patch, either on the loopback > > data rate at >16kB MTU, or on the myri. I'm keeping it at hand for > > experimentation anyway. > > >

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:52 +0100, Willy Tarreau wrote: > OK so I observed no change with this patch, either on the loopback > data rate at >16kB MTU, or on the myri. I'm keeping it at hand for > experimentation anyway. > Yeah, there was no bug. I rewrote it for net-next as a cleanup/optim only.

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > > > > > Ah interesting because these were some of th

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:32 +0100, Willy Tarreau wrote: > It's 0cf833ae (net: loopback: set default mtu to 64K). And I could > reproduce it with 3.6 by setting loopback's MTU to 65536 by hand. > The trick is that once the MTU has been set to this large a value, > even when I set it back to 16kB th

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:22:13PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 03:18 +0100, Willy Tarreau wrote: > > On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > > > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > > > On Sat, 2013-01-05 at 17:40 -0800, Eric Duma

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 03:18 +0100, Willy Tarreau wrote: > On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wr

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 06:16:31PM -0800, Eric Dumazet wrote: > On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > > > > > Ah interesting because these were some of th

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 17:51 -0800, Eric Dumazet wrote: > On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > > > Ah interesting because these were some of the mm patches that I had > > > tried to revert. > > > > Hmm, or we shoul

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 17:40 -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > > > Ah interesting because these were some of the mm patches that I had > > tried to revert. > > Hmm, or we should fix __skb_splice_bits() > > I'll send a patch. > Could you try t

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 02:30 +0100, Willy Tarreau wrote: > Ah interesting because these were some of the mm patches that I had > tried to revert. Hmm, or we should fix __skb_splice_bits() I'll send a patch. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body o

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 05:21:16PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 01:50 +0100, Willy Tarreau wrote: > > > Yes, I've removed all zero counters in this short view for easier > > reading (complete version appended at the end of this email). This > > was after around 140 GB were tr

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 01:50 +0100, Willy Tarreau wrote: > Yes, I've removed all zero counters in this short view for easier > reading (complete version appended at the end of this email). This > was after around 140 GB were transferred : OK I only wanted to make sure skb were not linearized in xm

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
On Sat, Jan 05, 2013 at 04:02:03PM -0800, Eric Dumazet wrote: > On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote: > > > > 2) Another possibility would be that Myri card/driver doesnt like very > > > well high order pages. > > > > It looks like it has not changed much since 3.6 :-/ I really

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sun, 2013-01-06 at 00:29 +0100, Willy Tarreau wrote: > > 2) Another possibility would be that Myri card/driver doesnt like very > > well high order pages. > > It looks like it has not changed much since 3.6 :-/ I really suspect > something is wrong with memory allocation. I have tried revertin

Re: Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
Hi Eric, On Sat, Jan 05, 2013 at 03:18:46PM -0800, Eric Dumazet wrote: > Hi Willy, another good finding during the week end ! ;) Yes, I wanted to experiment with TFO and stopped on this :-) > 1) This looks like interrupts are spreaded on multiple cpus, and this > give Out Of Order problems with

Re: Major network performance regression in 3.7

2013-01-05 Thread Eric Dumazet
On Sat, 2013-01-05 at 22:49 +0100, Willy Tarreau wrote: > Hi, > > I'm observing multiple apparently unrelated network performance > issues in 3.7, to the point that I'm doubting it comes from the > network stack. > > My setup involves 3 machines connected point-to-point with myri > 10GE NICs (the

Major network performance regression in 3.7

2013-01-05 Thread Willy Tarreau
Hi, I'm observing multiple apparently unrelated network performance issues in 3.7, to the point that I'm doubting it comes from the network stack. My setup involves 3 machines connected point-to-point with myri 10GE NICs (the middle machine has 2 NICs). The middle machine normally runs haproxy, t