On Tue, 2018-01-23 at 14:09 -0800, Ben Greear wrote:
> On 01/23/2018 02:07 PM, Eric Dumazet wrote:
> > On Tue, 2018-01-23 at 13:49 -0800, Ben Greear wrote:
> > > On 01/22/2018 10:16 AM, Eric Dumazet wrote:
> > > > On Mon, 2018-01-22 at 09:28 -0800, Ben Greear wrote:
> > > > > My test case is to have 6 processes each create 5000 TCP IPv4
> > > > > connections to each other
> > > > > on a system with 16GB RAM and send slow-speed data. This works fine
> > > > > on a 4.7 kernel, but
> > > > > will not work at all on a 4.13. The 4.13 first complains about
> > > > > running out of tcp memory,
> > > > > but even after forcing those values higher, the max connections we
> > > > > can get is around 15k.
> > > > >
> > > > > Both kernels have my out-of-tree patches applied, so it is possible
> > > > > it is my fault
> > > > > at this point.
> > > > >
> > > > > Any suggestions as to what this might be caused by, or if it is fixed
> > > > > in more recent kernels?
> > > > >
> > > > > I will start bisecting in the meantime...
> > > > >
> > > >
> > > > Hi Ben
> > > >
> > > > Unfortunately I have no idea.
> > > >
> > > > Are you using loopback flows, or have I misunderstood you ?
> > > >
> > > > How loopback connections can be slow-speed ?
> > > >
> > >
> > > Hello Eric, looks like it is one of your commits that causes the issue
> > > I see.
> > >
> > > Here are some more details on my specific test case I used to bisect:
> > >
> > > I have two ixgbe ports looped back, configured on same subnet, but with
> > > different IPs.
> > > Routing table rules, SO_BINDTODEVICE, binding to specific IPs on both
> > > client and server
> > > side let me send-to-self over the external looped cable.
> > >
> > > I have 2 mac-vlans on each physical interface.
> > >
> > > I created 5 server-side connections on one physical port, and two more on
> > > one of the mac-vlans.
> > >
> > > On the client-side, I create a process that spawns 5000 connections to
> > > the corresponding server side.
> > >
> > > End result is 25,000 connections on one pair of real interfaces, and
> > > 10,000 connections on the
> > > mac-vlan ports.
> > >
> > > In the passing case, I get very close to all 5000 connections on all
> > > endpoints quickly.
> > >
> > > In the failing case, I get a max of around 16k connections on the two
> > > physical ports. The two mac-vlans have 10k connections
> > > across them working reliably. It seems to be an issue with 'connect'
> > > failing.
> > >
> > > connect(2074, {sa_family=AF_INET, sin_port=htons(33012),
> > > sin_addr=inet_addr("10.1.1.5")}, 16) = -1 EINPROGRESS (Operation now in
> > > progress)
> > > socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 2075
> > > fcntl(2075, F_GETFD) = 0
> > > fcntl(2075, F_SETFD, FD_CLOEXEC) = 0
> > > setsockopt(2075, SOL_SOCKET, SO_BINDTODEVICE,
> > > "eth4\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
> > > setsockopt(2075, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> > > bind(2075, {sa_family=AF_INET, sin_port=htons(0),
> > > sin_addr=inet_addr("10.1.1.4")}, 16) = 0
> > > getsockopt(2075, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
> > > getsockopt(2075, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
> > > setsockopt(2075, SOL_TCP, TCP_NODELAY, [0], 4) = 0
> > > fcntl(2075, F_GETFL) = 0x2 (flags O_RDWR)
> > > fcntl(2075, F_SETFL, O_ACCMODE|O_NONBLOCK) = 0
> > > connect(2075, {sa_family=AF_INET, sin_port=htons(33012),
> > > sin_addr=inet_addr("10.1.1.5")}, 16) = -1 EINPROGRESS (Operation now in
> > > progress)
> > > socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 2076
> > > fcntl(2076, F_GETFD) = 0
> > > fcntl(2076, F_SETFD, FD_CLOEXEC) = 0
> > > setsockopt(2076, SOL_SOCKET, SO_BINDTODEVICE,
> > > "eth4\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
> > > setsockopt(2076, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> > > bind(2076, {sa_family=AF_INET, sin_port=htons(0),
> > > sin_addr=inet_addr("10.1.1.4")}, 16) = 0
> > > getsockopt(2076, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
> > > getsockopt(2076, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
> > > setsockopt(2076, SOL_TCP, TCP_NODELAY, [0], 4) = 0
> > > fcntl(2076, F_GETFL) = 0x2 (flags O_RDWR)
> > > fcntl(2076, F_SETFL, O_ACCMODE|O_NONBLOCK) = 0
> > > connect(2076, {sa_family=AF_INET, sin_port=htons(33012),
> > > sin_addr=inet_addr("10.1.1.5")}, 16) = -1 EADDRNOTAVAIL (Cannot assign
> > > requested address)
> > > ....
> > >
> > >
> > > ea8add2b190395408b22a9127bed2c0912aecbc8 is the first bad commit
> > > commit ea8add2b190395408b22a9127bed2c0912aecbc8
> > > Author: Eric Dumazet <eduma...@google.com>
> > > Date: Thu Feb 11 16:28:50 2016 -0800
> > >
> > > tcp/dccp: better use of ephemeral ports in bind()
> > >
> > > Implement strategy used in __inet_hash_connect() in opposite way :
> > >
> > > Try to find a candidate using odd ports, then fallback to even ports.
> > >
> > > We no longer disable BH for whole traversal, but one bucket at a
> > > time.
> > > We also use cond_resched() to yield cpu to other tasks if needed.
> > >
> > > I removed one indentation level and tried to mirror the loop we have
> > > in __inet_hash_connect() and variable names to ease code maintenance.
> > >
> > > Signed-off-by: Eric Dumazet <eduma...@google.com>
> > > Signed-off-by: David S. Miller <da...@davemloft.net>
> > >
> > > :040000 040000 3af4595c6eb6d331e1cba78a142d44e00f710d81
> > > e0c014ae8b7e2867256eff60f6210821d36eacef M net
> > >
> > >
> > > I will be happy to test patches or try to get any other results that
> > > might help diagnose
> > > this problem better.
> >
> > Problem is I do not see anything obvious here.
> >
> > Please provide /proc/sys/net/ipv4/ip_local_port_range
>
> [root@lf1003-e3v2-13100124-f20x64 ~]# cat
> /proc/sys/net/ipv4/ip_local_port_range
> 10000 61001
>
> >
> > Also you probably could use IP_BIND_ADDRESS_NO_PORT socket option
> > before the bind()
>
> I'll read up on that to see what it does...
man 7 ip
IP_BIND_ADDRESS_NO_PORT (since Linux 4.2)
Inform
the kernel to not reserve an ephemeral port
when using
bind(2) with a port number of 0. The
port will later be
automatically chosen at con‐
nect(2) time, in a way
that allows sharing a source
port as long as the 4-tuple
is unique.
