Hi!
Problem description:
In a customers network more than 2k clients connect to a server and
perform https requests. When in the morning more and more clients become
active, the number of connections rises until more and more clients fail
to connect to the server. The reason appears to be packet losses.
Question:
Are we hitting system limits or resource exhaustion that we should have
configured higher? Any other idea what to look for?
Thanks in advance!
T.
############################################################
Findings:
Debugging on the production server is not trivial, so we've done some
tests on the client side first and those showed that when there's an
error, the client sends a tcp SYN but does not receive back a SYN-ACK.
############################################################
Setup, OS:
We're using Bernsteins daemontools to start a go (golang) based https
server in a chroot running as user www on OpenBSD63.
hardware:
The server runs as VM on VMWare Esxi 6.5. The above mentioned client ran
on the same ESXi server, so there's only one virtual 10GB switch between
the client and the server. The issue has been reproduced when the server
was running on a different virtualisation host.
Network:
To redirect port 443 to the high port that the userspace golang code can
open, we're using pf:
port 443 -> pf -> 8443 -> go-httpd -> 127.0.0.1(Database)
pf.conf:
#[...]
block all
#[...]
pass in proto tcp from any to any port 443 rdr-to 127.0.0.1 port 8443
keep state
Limits:
User www is member of login-class daemon. The ulimits for daemon in
/etc/login.conf were set to
daemon:\
:ignorenologin:\
:datasize=infinity:\
:maxproc=infinity:\
:openfiles-max=8192:\
:openfiles-cur=1024:\
:stacksize-cur=8M:\
:localcipher=blowfish,a:\
:tc=default:
