Hi Eric
Thanks for your response. My apologies for being late in getting back to you -
I wasn't able to have access to the lab hardware on the weekend.
I performed your test as suggested - I've provided a side-by-side diff of the
nstat output below for the SCTP test only (not the TCP test). Note that the
fields that are output are somewhat different for the two kernels - i.e. some
fields exist in one but not the other (presumably this comes from the kernel
internals?).
Other than seeing 'larger' throughput numbers in this output I'm not sure what
to take from it - I'm certainly not a networking expert :-( Let me know if
there's anything that speaks to you.
Note that this test was again done on a clean, freshly rebooted and idle
system. Let me know if there's any issues with the output format of this data
in the email.
Thanks,
Peter
3.4.2 4.4.0
-------------------------------------------------------------------------------------------------------
IpInReceives 3457295 0.0 | IpInReceives
1151189 0.0
IpInDelivers 3457295 0.0 | IpInDelivers
1151189 0.0
IpOutRequests 6864955 0.0 | IpOutRequests
2249622 0.0
IcmpInErrors 158 0.0 | IcmpInErrors
159 0.0
IcmpInTimeExcds 152 0.0 | IcmpInTimeExcds
151 0.0
IcmpInEchoReps 6 0.0 | IcmpInEchoReps
8 0.0
IcmpOutErrors 158 0.0 | IcmpOutErrors
159 0.0
IcmpOutTimeExcds 152 0.0 | IcmpOutTimeExcds
151 0.0
IcmpOutTimestamps 6 0.0 | IcmpOutTimestamps
8 0.0
IcmpMsgInType3 152 0.0 | IcmpMsgInType3
151 0.0
IcmpMsgInType8 6 0.0 | IcmpMsgInType8
8 0.0
IcmpMsgOutType0 6 0.0 | IcmpMsgOutType0
8 0.0
IcmpMsgOutType3 152 0.0 | IcmpMsgOutType3
151 0.0
TcpActiveOpens 1 0.0 TcpActiveOpens
1 0.0
TcpPassiveOpens 3 0.0 | TcpPassiveOpens
4 0.0
TcpInSegs 70 0.0 | TcpInSegs
117 0.0
TcpOutSegs 66 0.0 | TcpOutSegs
110 0.0
| TcpOutRsts
24 0.0
UdpInDatagrams 608 0.0 | UdpInDatagrams
604 0.0
UdpNoPorts 152 0.0 | UdpNoPorts
151 0.0
UdpOutDatagrams 760 0.0 | UdpOutDatagrams
755 0.0
| UdpIgnoredMulti
144 0.0
TcpExtTW 2 0.0
TcpExtDelayedACKs 3 0.0 | TcpExtDelayedACKs
4 0.0
TcpExtTCPHPHits 25 0.0 | TcpExtTCPHPHits
41 0.0
TcpExtTCPPureAcks 12 0.0 | TcpExtTCPPureAcks
14 0.0
TcpExtTCPHPAcks 18 0.0 | TcpExtTCPHPAcks
26 0.0
| TcpExtTCPRcvCoalesce
12 0.0
| TcpExtTCPOrigDataSent
57 0.0
IpExtInBcastPkts 152 0.0 | IpExtInBcastPkts
144 0.0
IpExtInOctets 166191161 0.0 | IpExtInOctets
55395212 0.0
IpExtOutOctets 9107586685 0.0 | IpExtOutOctets
2983660504 0.0
IpExtInBcastOctets 37356 0.0 | IpExtInBcastOctets
35328 0.0
| IpExtInNoECTPkts
1175 0.0
| IpExtInECT0Pkts
1150014 0.0
-----Original Message-----
From: Eric Dumazet [mailto:eric.duma...@gmail.com]
Sent: April-18-16 8:17 AM
To: Butler, Peter <pbut...@sonusnet.com>
Cc: netdev@vger.kernel.org
Subject: Re: Poorer networking performance in later kernels?
On Fri, 2016-04-15 at 15:33 -0700, Eric Dumazet wrote:
> On Fri, 2016-04-15 at 21:02 +0000, Butler, Peter wrote:
> > (Please keep me CC'd to all comments/responses)
> >
> > I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop in
> > networking performance. Nothing was changed on the test systems, other
> > than the kernel itself (and kernel modules). The identical .config used to
> > build the 3.4.2 kernel was brought over into the 4.4.0 kernel source tree,
> > and any configuration differences (e.g. new parameters, etc.) were taken as
> > default values.
> >
> > The testing was performed on the same actual hardware for both kernel
> > versions (i.e. take the existing 3.4.2 physical setup, simply boot into the
> > (new) kernel and run the same test). The netperf utility was used for
> > benchmarking and the testing was always performed on idle systems.
> >
> > TCP testing yielded the following results, where the 4.4.0 kernel only got
> > about 1/2 of the throughput:
> >
> > Recv Send Send Utilization
> > Service Demand
> > Socket Socket Message Elapsed Send Recv
> > Send Recv
> > Size Size Size Time Throughput local remote
> > local remote
> > bytes bytes bytes secs. 10^6bits/s % S % S
> > us/KB us/KB
> >
> > 3.4.2 13631488 13631488 8952 30.01 9370.29 10.14 6.50
> > 0.709 0.454
> > 4.4.0 13631488 13631488 8952 30.02 5314.03 9.14 14.31
> > 1.127 1.765
> >
> > SCTP testing yielded the following results, where the 4.4.0 kernel only got
> > about 1/3 of the throughput:
> >
> > Recv Send Send Utilization
> > Service Demand
> > Socket Socket Message Elapsed Send Recv
> > Send Recv
> > Size Size Size Time Throughput local remote
> > local remote
> > bytes bytes bytes secs. 10^6bits/s % S % S
> > us/KB us/KB
> >
> > 3.4.2 13631488 13631488 8952 30.00 2306.22 13.87 13.19
> > 3.941 3.747
> > 4.4.0 13631488 13631488 8952 30.01 882.74 16.86 19.14
> > 12.516 14.210
> >
> > The same tests were performed a multitude of time, and are always
> > consistent (within a few percent). I've also tried playing with various
> > run-time kernel parameters (/proc/sys/kernel/net/...) on the 4.4.0 kernel
> > to alleviate the issue but have had no success at all.
> >
> > I'm at a loss as to what could possibly account for such a discrepancy...
>
> Maybe new kernel is faster and you have drops somewhere ?
>
> nstat >/dev/null
> netperf -H ...
> nstat
>
> Would help
>
Are you receiving my mails, or simply ignoring them ?
Thanks.
