Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-13 Thread Barney Cordoba
:* Rick Macklem >> *Cc:* FreeBSD Net ; David Wolfskill >> *Sent:* Wednesday, September 4, 2013 11:26 AM >> *Subject:* TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) >> >> On 9/4/2013 8:50 AM, Rick Macklem wrote: >>> David Wolfskill wrote: >>

Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-11 Thread Mark Felder
On Tue, Sep 10, 2013, at 18:11, Mike Tancsa wrote: > > IIRC, the new iSCSI stack is currently tested more for correctness than > performance? > Yes, but it's in the kernel vs istgt which is all userland code. It could very well be faster. I'm quite interested in seeing someone put up some seriou

Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-10 Thread Mike Tancsa
On 9/10/2013 7:04 PM, Rick Macklem wrote: > Mike Tancsa wrote: >> On 9/10/2013 6:42 PM, Barney Cordoba wrote: >>> NFS has been broken since Day 1, so lets not come to conclusions >>> about >>> anything >>> as it relates to NFS. >> >> iSCSI is NFS ? >> > It would be really nice if you could try tras

Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-10 Thread Rick Macklem
gt; *To:* Rick Macklem > > *Cc:* FreeBSD Net ; David Wolfskill > > > > *Sent:* Wednesday, September 4, 2013 11:26 AM > > *Subject:* TSO help or hindrance ? (was Re: TSO and FreeBSD vs > > Linux) > > > > On 9/4/2013 8:50 AM, Rick Macklem wrote: > >> Dav

Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-10 Thread Mike Tancsa
> *From:* Mike Tancsa > *To:* Rick Macklem > *Cc:* FreeBSD Net ; David Wolfskill > *Sent:* Wednesday, September 4, 2013 11:26 AM > *Subject:* TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) > > On 9/4/2013 8:50 AM, Rick Macklem wrote: >> David Wolfskill wrote:

Re: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-10 Thread Barney Cordoba
Re: TSO and FreeBSD vs Linux) On 9/4/2013 8:50 AM, Rick Macklem wrote: > David Wolfskill wrote: >> >> >> I noticed that when I tried to write files to NFS, I could write >> small >> files OK, but larger ones seemed to ... hang. >> * "ifconfig -v e

Re: TSO and FreeBSD vs Linux

2013-09-05 Thread David Wolfskill
On Wed, Sep 04, 2013 at 03:52:19PM -0700, Adrian Chadd wrote: > .. can you use -z to clear the stats out just before you do a test? > > > -adrian > > > On 4 September 2013 15:35, David Wolfskill wrote: > > > On Wed, Sep 04, 2013 at 03:25:59PM -0700, Adrian Chadd wrote: > > > What's netstat -s

Re: TSO and FreeBSD vs Linux

2013-09-04 Thread Adrian Chadd
Hiya, David - can you put together a minimal test case that others can reproduce? I have a bunch of gige intel NICs that I can try this with when I'm back in the office. Thanks, -adrian On 4 September 2013 05:50, Rick Macklem wrote: > David Wolfskill wrote: > > On Wed, Aug 21, 2013 at 07:1

Re: TSO and FreeBSD vs Linux

2013-09-04 Thread David Wolfskill
On Wed, Sep 04, 2013 at 03:25:59PM -0700, Adrian Chadd wrote: > What's netstat -sp tcp show before/after the test? > ... Typescript attached. Peace, david -- David H. Wolfskill da...@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old gi

Re: TSO and FreeBSD vs Linux

2013-09-04 Thread Adrian Chadd
What's netstat -sp tcp show before/after the test? -adrian On 4 September 2013 15:20, David Wolfskill wrote: > On Wed, Sep 04, 2013 at 03:03:12PM -0700, Adrian Chadd wrote: > > Hiya, > > > > David - can you put together a minimal test case that others can > reproduce? > > I have a bunch of g

Re: TSO and FreeBSD vs Linux

2013-09-04 Thread David Wolfskill
On Wed, Sep 04, 2013 at 03:03:12PM -0700, Adrian Chadd wrote: > Hiya, > > David - can you put together a minimal test case that others can reproduce? > I have a bunch of gige intel NICs that I can try this with when I'm back in > the office. > ... On NFS client machine (w/ Intel NICs): * Ensure t

TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux)

2013-09-04 Thread Mike Tancsa
On 9/4/2013 8:50 AM, Rick Macklem wrote: > David Wolfskill wrote: >> >> >> I noticed that when I tried to write files to NFS, I could write >> small >> files OK, but larger ones seemed to ... hang. >> * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. >> * "sysctl net.inet.tcp.tso" showe

Re: TSO and FreeBSD vs Linux

2013-09-04 Thread Rick Macklem
David Wolfskill wrote: > On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote: > > On 13.08.2013 19:29, Julian Elischer wrote: > > > I have been tracking down a performance embarrassment on AMAZON > > > EC2 and have found it I think. > > > Our OS cousins over at Linux land have implement

Re: TSO and FreeBSD vs Linux

2013-09-03 Thread Hooman Fazaeli
On 9/4/2013 9:23 AM, Julian Elischer wrote: On 9/4/13 6:49 AM, David Wolfskill wrote: On Tue, Sep 03, 2013 at 12:27:34PM -0700, David Wolfskill wrote: ... As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked without a hitch or a whine. And I was able to copy all 117709618 bytes, n

Re: TSO and FreeBSD vs Linux

2013-09-03 Thread Julian Elischer
On 9/4/13 6:49 AM, David Wolfskill wrote: On Tue, Sep 03, 2013 at 12:27:34PM -0700, David Wolfskill wrote: ... As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked without a hitch or a whine. And I was able to copy all 117709618 bytes, not just 2097152 (2^21). The above command sh

Re: TSO and FreeBSD vs Linux

2013-09-03 Thread Michael Sierchio
I must have discovered this and forgotten - all my AMIs have net.inet.tcp.tso=0 dev.xn.0.enable_lro=0 or ifconfig xn0 -tso -lro - M On Tue, Sep 3, 2013 at 4:56 PM, Adrian Chadd wrote: > ... this is bad behaviour. So yes, it needs to be chased up and repaired. > > Thanks for finding it out!

Re: TSO and FreeBSD vs Linux

2013-09-03 Thread David Wolfskill
On Tue, Sep 03, 2013 at 12:27:34PM -0700, David Wolfskill wrote: > ... > As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked without > a hitch or a whine. And I was able to copy all 117709618 bytes, not just > 2097152 (2^21). The above command should (of course) have read s

Re: TSO and FreeBSD vs Linux

2013-09-03 Thread Adrian Chadd
... this is bad behaviour. So yes, it needs to be chased up and repaired. Thanks for finding it out! On 3 September 2013 12:27, David Wolfskill wrote: > On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote: > > On 13.08.2013 19:29, Julian Elischer wrote: > > > I have been tracking d

Re: TSO and FreeBSD vs Linux

2013-09-03 Thread David Wolfskill
On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote: > On 13.08.2013 19:29, Julian Elischer wrote: > > I have been tracking down a performance embarrassment on AMAZON EC2 and > > have found it I think. > > Our OS cousins over at Linux land have implemented some interesting > > behavio

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-26 Thread Andre Oppermann
On 26.08.2013 16:46, Luigi Rizzo wrote: On Mon, Aug 26, 2013 at 04:27:59PM +0200, Andre Oppermann wrote: ... 1. lle lock to rmlock. 2. if_addr and IN_ADDR locks to rmlocks. 3. routing table locking (rmlocks, and by doing away with rtentry locks and refcounting through copy-out on lookup

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-26 Thread Luigi Rizzo
On Mon, Aug 26, 2013 at 04:27:59PM +0200, Andre Oppermann wrote: ... > > 1. lle lock to rmlock. > > 2. if_addr and IN_ADDR locks to rmlocks. > > 3. routing table locking (rmlocks, and by doing away with rtentry locks and > refcounting > through copy-out on lookup and prohibition of having a

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-26 Thread Andre Oppermann
On 25.08.2013 16:42, Adrian Chadd wrote: On 24 August 2013 10:09, Andre Oppermann mailto:an...@freebsd.org>> wrote: On 24.08.2013 19:04, Adrian Chadd wrote: I'm very close to starting an mbuf batching thing to use in a few places like receive, transmit and transmit

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-25 Thread Adrian Chadd
On 24 August 2013 10:09, Andre Oppermann wrote: > On 24.08.2013 19:04, Adrian Chadd wrote: > >> I'm very close to starting an mbuf batching thing to use in a few places >> like receive, transmit and >> transmit completion -> free path. I'd be interested in your >> review/feedback and testing as i

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-24 Thread Andre Oppermann
On 24.08.2013 19:04, Adrian Chadd wrote: I'm very close to starting an mbuf batching thing to use in a few places like receive, transmit and transmit completion -> free path. I'd be interested in your review/feedback and testing as it sounds like something you can easily stress test there. :)

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-24 Thread Adrian Chadd
Hi! Can you give us actual numbers? Something that we can easily graph? Do you have a diff against -HEAD? I'd like to stare at it a bunch and think about merging some stuff in. :-) How are you testing it? Is it something I can set up in our lab and thoroughly thrash? I'm very close to starting

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-24 Thread Alexander V. Chernikov
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 22.08.2013 00:51, Andre Oppermann wrote: > On 19.08.2013 13:42, Alexander V. Chernikov wrote: >> On 14.08.2013 19:48, Luigi Rizzo wrote: >>> On Wed, Aug 14, 2013 at 05:40:28PM +0200, Marko Zec wrote: On Wednesday 14 August 2013 14:40:24 Luigi R

Re: TSO and FreeBSD vs Linux

2013-08-23 Thread Julian Elischer
On 8/22/13 1:12 AM, Andre Oppermann wrote: On 13.08.2013 19:29, Julian Elischer wrote: I have been tracking down a performance embarrassment on AMAZON EC2 and have found it I think. Our OS cousins over at Linux land have implemented some interesting behaviour when TSO is in use. There used to

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-21 Thread Andre Oppermann
On 19.08.2013 13:42, Alexander V. Chernikov wrote: On 14.08.2013 19:48, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 05:40:28PM +0200, Marko Zec wrote: On Wednesday 14 August 2013 14:40:24 Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: ... FWIW, appa

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-21 Thread Barney Cordoba
From: Andre Oppermann To: Adrian Chadd Cc: Barney Cordoba ; Luigi Rizzo ; "freebsd-net@freebsd.org" Sent: Wednesday, August 21, 2013 2:19 PM Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) On 18.08.201

Re: TSO and FreeBSD vs Linux

2013-08-21 Thread Andre Oppermann
On 15.08.2013 01:27, Kevin Oberman wrote: On Wed, Aug 14, 2013 at 12:46 PM, Julian Elischer wrote: On 8/14/13 3:23 PM, Lawrence Stewart wrote: On 08/14/13 16:33, Julian Elischer wrote: They switched to using an initial window of 10 segments some time ago. FreeBSD starts with 3 or more rec

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-21 Thread Andre Oppermann
On 14.08.2013 12:21, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 05:23:02PM +1000, Lawrence Stewart wrote: I think (check the driver code in question as I'm not sure) that if you "ifconfig lro" and the driver has hardware support or has been made aware of our software implementation, it should D

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-21 Thread Andre Oppermann
On 18.08.2013 23:54, Adrian Chadd wrote: Hi, I think the "UNIX architecture" is a bit broken for anything other than the occasional (for various traffic levels defining "occasional!") traffic connection. It's serving us well purely through the sheer force of will of modern CPU power but I think

Re: TSO and FreeBSD vs Linux

2013-08-21 Thread Andre Oppermann
On 13.08.2013 19:29, Julian Elischer wrote: I have been tracking down a performance embarrassment on AMAZON EC2 and have found it I think. Our OS cousins over at Linux land have implemented some interesting behaviour when TSO is in use. There used to be a different problem with EC2 and FreeBS

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-19 Thread Barney Cordoba
From: Luigi Rizzo To: Barney Cordoba Cc: "freebsd-net@freebsd.org" ; Adrian Chadd Sent: Sunday, August 18, 2013 5:16 PM Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) On Sun, Aug 18, 2013 at 11:01 PM,

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-19 Thread Alexander V. Chernikov
On 14.08.2013 19:48, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 05:40:28PM +0200, Marko Zec wrote: On Wednesday 14 August 2013 14:40:24 Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: ... FWIW, apparently we already have that infrastrucure in place -

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-19 Thread Alexander V. Chernikov
On 14.08.2013 19:40, Marko Zec wrote: On Wednesday 14 August 2013 14:40:24 Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: On 14.08.2013 16:05, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: Hello, Luigi. You wrot

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-19 Thread Alexander V. Chernikov
On 14.08.2013 18:00, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 05:01:05PM +0400, Alexander V. Chernikov wrote: On 14.08.2013 16:40, Luigi Rizzo wrote: ... You can save rte&arp, however doing this gives you perfect chance to crash your kernel if egress interface is destroyed (like vlan or ng o

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Ian Smith
On Sun, 18 Aug 2013 14:03:27 -0700, Barney Cordoba wrote: > Criticism is the bedrock of innovation. Constructive criticism, with clear design even without code, can be. Relentless negativity achieves nothing, and fails to compile. Ian ___ freebsd-net

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Jim Thompson
On Aug 18, 2013, at 4:16 PM, Luigi Rizzo wrote: > The mistake, i think, > is to expect that there is one magic solution to handle all the useful > cases. AKA: not all the world is Yahoo. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Adrian Chadd
Hi, I think the "UNIX architecture" is a bit broken for anything other than the occasional (for various traffic levels defining "occasional!") traffic connection. It's serving us well purely through the sheer force of will of modern CPU power but I think we can do a lot better. _I_ think the corr

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Luigi Rizzo
On Sun, Aug 18, 2013 at 11:01 PM, Barney Cordoba wrote: > That's fine, it's a test tool, not a solution. It just seems that it gets > pushed as if it's some sort of real > world solution, which it's not. The idea that bringing packets into user > space to forward them rather > than just replacing

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Barney Cordoba
Criticism is the bedrock of innovation. From: Vijay Singh To: Barney Cordoba Cc: Adrian Chadd ; "freebsd-net@freebsd.org" Sent: Sunday, August 18, 2013 3:46 PM Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Barney Cordoba
an a kludge in the first place. BC From: Adrian Chadd To: Barney Cordoba Cc: "freebsd-net@freebsd.org" Sent: Sunday, August 18, 2013 3:18 PM Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) On 18 Aug

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Vijay Singh
> netmap. > > > > From: Adrian Chadd > To: Jim Thompson > Cc: Barney Cordoba ; FreeBSD Net > ; Luigi Rizzo ; Lawrence Stewart > > Sent: Sunday, August 18, 2013 11:57 AM > Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs > Lin

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Adrian Chadd
On 18 August 2013 11:39, Barney Cordoba wrote: > Great. Never has the been a better explanation for the word Kludge than > netmap. > Nah. Netmap is a reimplementation of some reasonably well known ways of pushing bits. Luigi just pushed it up to eleven and demonstrated what current hardware is c

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Barney Cordoba
, not ack coalescing (Re: TSO and FreeBSD vs Linux) Right. Well, post some profiling data, let's figure this out sometime. Luigi can do bridging with 2 cores using netmap. So it's technically possible. There's just a lot of kernel gunk in the way of doing it ye olde way. -adrian

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Adrian Chadd
Right. Well, post some profiling data, let's figure this out sometime. Luigi can do bridging with 2 cores using netmap. So it's technically possible. There's just a lot of kernel gunk in the way of doing it ye olde way. -adrian On 18 August 2013 07:25, Jim Thompson wrote: > > On Aug 18, 201

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Jim Thompson
On Aug 18, 2013, at 8:48 AM, Barney Cordoba wrote: > I could fill a tx queue with 10gb of traffic with yesteryear's cpus. It's > not an achievement. Being able to bridge > real traffic at 10gb/s with 2 cores is Or forward at layer 3. Or filter packets. Or IPSEC. Or... _

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-18 Thread Barney Cordoba
From: Adrian Chadd To: Barney Cordoba Cc: Luigi Rizzo ; Lawrence Stewart ; FreeBSD Net Sent: Saturday, August 17, 2013 11:59 AM Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) ... we get perfectly good throu

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-17 Thread Adrian Chadd
... we get perfectly good throughput without 400k ints a second on the ixgbe driver. As in, I can easily saturate 2 x 10GE on ixgbe hardware with a handful of flows. That's not terribly difficult. However, there's a few interesting problems that need addressing: * There's lock contention between

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-17 Thread Barney Cordoba
ut, not ack coalescing (Re: TSO and FreeBSD vs Linux) On Wed, Aug 14, 2013 at 05:23:02PM +1000, Lawrence Stewart wrote: > On 08/14/13 16:33, Julian Elischer wrote: > > On 8/14/13 11:39 AM, Lawrence Stewart wrote: > >> On 08/14/13 03:29, Julian Elischer wrote: > >>>

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-17 Thread Barney Cordoba
ssue. BC From: Luigi Rizzo To: Lawrence Stewart Cc: FreeBSD Net Sent: Wednesday, August 14, 2013 6:21 AM Subject: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) On Wed, Aug 14, 2013 at 05:23:02PM +1000, Lawrence Stewart wrote: > On 08/14/13 16:3

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-16 Thread Adrian Chadd
On 16 August 2013 10:06, Luigi Rizzo wrote: > the usenix paper and talk > > https://www.usenix.org/conference/usenixfederatedconferencesweek/netmap-novel-framework-fast-packet-io > should give a short description of what i did. > > Basically i manually insert returns (and free mbufs and resources

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-16 Thread Luigi Rizzo
the usenix paper and talk https://www.usenix.org/conference/usenixfederatedconferencesweek/netmap-novel-framework-fast-packet-io should give a short description of what i did. Basically i manually insert returns (and free mbufs and resources) in some place in the path and then compute the time con

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-16 Thread Adrian Chadd
Luigi, Did you ever publish your patches or methodology for doing sampling? -adrian On 14 August 2013 21:00, Vijay Singh wrote: > Is that what FLOWTABLE does? Also we need a mechanism to record time spent > at various layers in the stack. Luigi has used his own methods but we're > lacking s

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-16 Thread Julian Elischer
On 8/14/13 6:21 PM, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 05:23:02PM +1000, Lawrence Stewart wrote: On 08/14/13 16:33, Julian Elischer wrote: On 8/14/13 11:39 AM, Lawrence Stewart wrote: On 08/14/13 03:29, Julian Elischer wrote: I have been tracking down a performance embarrassment on AM

Re: TSO and FreeBSD vs Linux

2013-08-15 Thread Julian Elischer
On 8/15/13 7:27 AM, Kevin Oberman wrote: On Wed, Aug 14, 2013 at 12:46 PM, Julian Elischer mailto:jul...@freebsd.org>> wrote: so I ran on 9.2-beta ( a week or two old) and it had similar problems.. only worse.. 9.2 actually sends multiple packets when is doesn't need to.. h

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-14 Thread Vijay Singh
Is that what FLOWTABLE does? Also we need a mechanism to record time spent at various layers in the stack. Luigi has used his own methods but we're lacking something more generic. At work we have some crude tools that use mcount information to indirectly measure costs but they are not reliable a

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-14 Thread Luigi Rizzo
On Wed, Aug 14, 2013 at 12:40:19PM -0700, Peter Wemm wrote: > On Wed, Aug 14, 2013 at 11:11 AM, Adrian Chadd wrote: > > On 14 August 2013 04:47, Lev Serebryakov wrote: > > > > > >> And we should invalidate this info on ARP/route changes, or connection > >> will be lost in such cases, am I righ

Re: TSO and FreeBSD vs Linux

2013-08-14 Thread Kevin Oberman
On Wed, Aug 14, 2013 at 12:46 PM, Julian Elischer wrote: > On 8/14/13 3:23 PM, Lawrence Stewart wrote: > >> On 08/14/13 16:33, Julian Elischer wrote: >> >> They switched to using an initial window of 10 segments some time ago. FreeBSD starts with 3 or more recently, 10 if you're running rece

Re: TSO and FreeBSD vs Linux

2013-08-14 Thread Julian Elischer
On 8/14/13 3:23 PM, Lawrence Stewart wrote: On 08/14/13 16:33, Julian Elischer wrote: They switched to using an initial window of 10 segments some time ago. FreeBSD starts with 3 or more recently, 10 if you're running recent 9-STABLE or 10-CURRENT. I tried setting initial values as shown: n

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-14 Thread Peter Wemm
On Wed, Aug 14, 2013 at 11:11 AM, Adrian Chadd wrote: > On 14 August 2013 04:47, Lev Serebryakov wrote: > > >> And we should invalidate this info on ARP/route changes, or connection >> will be lost in such cases, am I right?.. So, on each such event code >> should look into all sockets and ch

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-14 Thread Adrian Chadd
On 14 August 2013 04:47, Lev Serebryakov wrote: > And we should invalidate this info on ARP/route changes, or connection > will be lost in such cases, am I right?.. So, on each such event code > should look into all sockets and check, if routing/ARP information is > still > valid for them.

Re: TSO and FreeBSD vs Linux

2013-08-14 Thread Julian Elischer
On 8/14/13 2:33 PM, Julian Elischer wrote: On 8/14/13 11:39 AM, Lawrence Stewart wrote: There's a thing controlled by ethtool called GRO (generic receive offload) which appears to be enabled by default on at least Ubuntu and I guess other Linux's too. It's responsible for aggregating ACKs and

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Luigi Rizzo
On Wed, Aug 14, 2013 at 05:40:28PM +0200, Marko Zec wrote: > On Wednesday 14 August 2013 14:40:24 Luigi Rizzo wrote: > > On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: ... > FWIW, apparently we already have that infrastrucure in place - if_rele() > calls if_free_internal()

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Marko Zec
On Wednesday 14 August 2013 14:40:24 Luigi Rizzo wrote: > On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: > > On 14.08.2013 16:05, Luigi Rizzo wrote: > > > On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: > > >> Hello, Luigi. > > >> You wrote 14 ?

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Luigi Rizzo
On Wed, Aug 14, 2013 at 05:01:05PM +0400, Alexander V. Chernikov wrote: > On 14.08.2013 16:40, Luigi Rizzo wrote: ... > >> You can save rte&arp, however doing this > >> gives you perfect chance to crash your kernel if egress interface is > >> destroyed (like vlan or ng or tun). > > I hope I learned

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Alexander V. Chernikov
On 14.08.2013 16:40, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: On 14.08.2013 16:05, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: Hello, Luigi. You wrote 14 ?? 2013 ??., 14:21:09: LR> Then the p

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Luigi Rizzo
On Wed, Aug 14, 2013 at 04:15:25PM +0400, Alexander V. Chernikov wrote: > On 14.08.2013 16:05, Luigi Rizzo wrote: > > On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: > >> Hello, Luigi. > >> You wrote 14 ?? 2013 ??., 14:21:09: > >> > >> LR> Then the problem remains that

Re: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Alexander V. Chernikov
On 14.08.2013 16:05, Luigi Rizzo wrote: On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: Hello, Luigi. You wrote 14 ?? 2013 ??., 14:21:09: LR> Then the problem remains that we should keep a copy of route and LR> arp information in the socket instead of redoing the lo

route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux))

2013-08-14 Thread Luigi Rizzo
On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: > Hello, Luigi. > You wrote 14 ?? 2013 ??., 14:21:09: > > LR> Then the problem remains that we should keep a copy of route and > LR> arp information in the socket instead of redoing the lookups on > LR> every single trans

Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-14 Thread Lev Serebryakov
Hello, Luigi. You wrote 14 августа 2013 г., 14:21:09: LR> Then the problem remains that we should keep a copy of route and LR> arp information in the socket instead of redoing the lookups on LR> every single transmission, as they consume some 25% of the time of LR> a sendto(), and probably even mo

it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)

2013-08-14 Thread Luigi Rizzo
On Wed, Aug 14, 2013 at 05:23:02PM +1000, Lawrence Stewart wrote: > On 08/14/13 16:33, Julian Elischer wrote: > > On 8/14/13 11:39 AM, Lawrence Stewart wrote: > >> On 08/14/13 03:29, Julian Elischer wrote: > >>> I have been tracking down a performance embarrassment on AMAZON EC2 and > >>> have foun

Re: TSO and FreeBSD vs Linux

2013-08-14 Thread Lawrence Stewart
On 08/14/13 16:33, Julian Elischer wrote: > On 8/14/13 11:39 AM, Lawrence Stewart wrote: >> On 08/14/13 03:29, Julian Elischer wrote: >>> I have been tracking down a performance embarrassment on AMAZON EC2 and >>> have found it I think. >> Let us please avoid conflating performance with throughput.

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Julian Elischer
On 8/14/13 11:39 AM, Lawrence Stewart wrote: On 08/14/13 03:29, Julian Elischer wrote: I have been tracking down a performance embarrassment on AMAZON EC2 and have found it I think. Let us please avoid conflating performance with throughput. The behaviour you go on to describe as a performance

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Lawrence Stewart
On 08/14/13 03:29, Julian Elischer wrote: > I have been tracking down a performance embarrassment on AMAZON EC2 and > have found it I think. Let us please avoid conflating performance with throughput. The behaviour you go on to describe as a performance embarrassment is actually a throughput diffe

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Navdeep Parhar
On 08/13/13 17:51, Julian Elischer wrote: > On 8/14/13 1:37 AM, Navdeep Parhar wrote: >> On 08/13/13 10:29, Julian Elischer wrote: >> .. >>> Has anyone done any work on aggregating ACKs, or delaying responding to >>> them? >> If LRO is enabled on the FreeBSD receiver, ACKs are already aggregated >>

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Julian Elischer
On 8/14/13 1:37 AM, Navdeep Parhar wrote: On 08/13/13 10:29, Julian Elischer wrote: .. Has anyone done any work on aggregating ACKs, or delaying responding to them? If LRO is enabled on the FreeBSD receiver, ACKs are already aggregated (a duplicate ACK will result in an immediate flush though.)

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Luigi Rizzo
On Tue, Aug 13, 2013 at 7:37 PM, Navdeep Parhar wrote: > On 08/13/13 10:29, Julian Elischer wrote: > .. > > > > Has anyone done any work on aggregating ACKs, or delaying responding to > > them? > > If LRO is enabled on the FreeBSD receiver, ACKs are already aggregated > (a duplicate ACK will resu

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Julian Elischer
On 8/14/13 1:29 AM, Julian Elischer wrote: I have been tracking down a performance embarrassment on AMAZON EC2 and have found it I think. Our OS cousins over at Linux land have implemented some interesting behaviour when TSO is in use. They seem to aggregate ACKS when there is a lot of traffic

Re: TSO and FreeBSD vs Linux

2013-08-13 Thread Navdeep Parhar
On 08/13/13 10:29, Julian Elischer wrote: .. > > Has anyone done any work on aggregating ACKs, or delaying responding to > them? If LRO is enabled on the FreeBSD receiver, ACKs are already aggregated (a duplicate ACK will result in an immediate flush though.) See tcp_lro_rx. Regards, Navdeep __

TSO and FreeBSD vs Linux

2013-08-13 Thread Julian Elischer
I have been tracking down a performance embarrassment on AMAZON EC2 and have found it I think. Our OS cousins over at Linux land have implemented some interesting behaviour when TSO is in use. They seem to aggregate ACKS when there is a lot of traffic so that they can create the largest possib