On Fri, Aug 30, 2019 at 4:54 AM Eric Dumazet <eric.duma...@gmail.com> wrote:
>
>
>
> On 8/29/19 9:26 PM, Willem de Bruijn wrote:
>
> > SO_REUSEPORT was not intended to be used in this way. Opening
> > multiple connected sockets with the same local port.
> >
> > But since the interface allowed connect after joining a group, and
> > that is being used, I guess that point is moot. Still, I'm a bit
> > surprised that it ever worked as described.
> >
> > Also note that the default distribution algorithm is not round robin
> > assignment, but hash based. So multiple consecutive datagrams arriving
> > at the same socket is not unexpected.
> >
> > I suspect that this quick hack might "work". It seemed to on the
> > supplied .c file:
> >
> > score = compute_score(sk, net, saddr, sport,
> > daddr, hnum, dif, sdif);
> > if (score > badness) {
> > - if (sk->sk_reuseport) {
> > + if (sk->sk_reuseport && !sk->sk_state !=
> > TCP_ESTABLISHED) {
This won't work for a mix of connected and connectionless sockets, of
course (even ignoring the typo), as it only skips reuseport on the
connected sockets.
> >
> > But a more robust approach, that also works on existing kernels, is to
> > swap the default distribution algorithm with a custom BPF based one (
> > SO_ATTACH_REUSEPORT_EBPF).
> >
>
> Yes, I suspect that reuseport could still be used by to load-balance incoming
> packets
> targetting the same 4-tuple.
>
> So all sockets would have the same score, and we would select the first
> socket in
> the list (if not applying reuseport hashing)
Can you elaborate a bit?
One option I see is to record in struct sock_reuseport if any port in
the group is connected and, if so, don't return immediately on the
first reuseport_select_sock hit, but continue the search for a higher
scoring connected socket.
Or do return immediately, but do this refined search in
reuseport_select_sock itself, as it has a reference to all sockets in the
group in sock_reuseport->socks[]. Instead of the straightforward hash.
Steve, Re: your point on a scalable QUIC server. That is an
interesting case certainly. Opening a connected socket per flow adds
both memory and port table pressure. I once looked into an SO_TXONLY
udp socket option that does not hash connected sockets into the port
table. In effect receiving on a small set of listening sockets (e.g.,
one per cpu) and sending over separate tx-only sockets. That still
introduces unnecessary memory allocation. OTOH it amortizes some
operations, such as route lookup.
Anyway, that does not fix the immediate issue you reported when using
SO_REUSEPORT as described.
