Hi Ihar,
On 4/3/24 4:37 PM, Ihar Hrachyshka wrote:
On Wed, Apr 3, 2024 at 4:15 PM Brian Haley via discuss
<ovs-discuss@openvswitch.org <mailto:ovs-discuss@openvswitch.org>> wrote:
Hi,
I recently have been seeing issues in a large environment where the
listen backlog of ovsdb-server, both NB and SB, is getting over-flowed,
for example:
17842 times the listen queue of a socket overflowed
17842 SYNs to LISTEN sockets dropped
There is more on NB than SB, but I was surprised to see any. I can only
guess at the moment it is happening when the leader changes and
hundreds
of nodes try and reconnect.
Looking at their sockets I can see the backlog is only set to 10:
$ ss -ltm | grep 664
LISTEN 0 10 0.0.0.0:6641 <http://0.0.0.0:6641> 0.0.0.0:*
LISTEN 0 10 0.0.0.0:6642 <http://0.0.0.0:6642> 0.0.0.0:*
Digging into the code, there is only two places where listen() is
called, one being inet_open_passive():
/* Listen. */
if (style == SOCK_STREAM && listen(fd, 10) < 0) {
error = sock_errno();
VLOG_ERR("%s: listen: %s", target, sock_strerror(error));
goto error;
}
There is no way to config around this to even test if increasing would
help in a running environment.
So my question is two-fold:
1) Should this be increased? 128, 256, 1024? I can send a patch.
2) Should this be configurable?
Has anyone else seen this?
Yes, I've seen it, though I was focusing on AF_UNIX behavior. You may
want to check this series:
https://patchwork.ozlabs.org/project/openvswitch/list/?series=382739&state=%2A&archive=both <https://patchwork.ozlabs.org/project/openvswitch/list/?series=382739&state=%2A&archive=both>
Specifically, it includes a patch to bump the value for INET sockets to
64:
https://patchwork.ozlabs.org/project/openvswitch/patch/20231118010703.4154866-2-ihrac...@redhat.com/ <https://patchwork.ozlabs.org/project/openvswitch/patch/20231118010703.4154866-2-ihrac...@redhat.com/>
The series includes a number of other fixes related to unixctl socket
handling; I am planning to revive this series in next few months, but
perhaps we could untangle this bump limit patch from the series and
merge it independently.
As to the question of potential configurability, I have nothing to
contribute.
Thanks for the quick response Ihar, I had actually missed the python
code as I only looked in *.c. I can see if bumping to 64 helps, I just
don't know how quickly it will be deployed and tested.
Regarding merging it independently, it seems from looking at the series
it might be something that could/should be backported in it's entirety.
If not I can try and break-out that one patch for previous versions.
And regarding configurability I can only say it would have been good to
have it to debug this easily, as opposed to re-building things.
-Brian
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
