Hi,
We are letting HAProxy pick what it think is best for our hardware. It
seems to always pick `nbproc = 1` (we only have one cpu socket), and
`nbthread = $(nproc)`
On 14/12/22 3:15, Emerson Gomes wrote:
Hi,
Have you tried increasing the number of processes/threads?
I dont see any nbthreads or nbproc in your config.
Check out https://www.haproxy.com/blog/multithreading-in-haproxy/
BR.,
Emerson
Em seg., 12 de dez. de 2022 às 02:49, Iago Alonso
<iago.alo...@marfeel.com> escreveu:
Hello,
We are performing a lot of load tests, and we hit what we think is an
artificial limit of some sort, or a parameter that we are not taking
into account (HAProxy config setting, kernel parameter…). We are
wondering if there’s a known limit on what HAProxy is able to process,
or if someone has experienced something similar, as we are thinking
about moving to bigger servers, and we don’t know if we will observe a
big difference.
When trying to perform the load test in production, we observe that we
can sustain 200k connections, and 10k rps, with a load1 of about 10.
The maxsslrate and maxsslconn are maxed out, but we handle the
requests fine, and we don’t return 5xx. Once we increase the load just
a bit and hit 11k rps and about 205k connections, we start to return
5xx and we rapidly decrease the load, as these are tests against
production.
Production server specs:
CPU: AMD Ryzen 7 3700X 8-Core Processor (16 threads)
RAM: DDR4 64GB (2666 MT/s)
When trying to perform a load test with synthetic tests using k6 as
our load generator against staging, we are able to sustain 750k
connections, with 20k rps. The load generator has a ramp-up time of
120s to achieve the 750k connections, as that’s what we are trying to
benchmark.
Staging server specs:
CPU: AMD Ryzen 5 3600 6-Core Processor (12 threads)
RAM: DDR4 64GB (3200 MT/s)
I've made a post about this on discourse, and I got the suggestion to
post here. In said post, I've included screenshots of some of our
Prometheus metrics.
https://discourse.haproxy.org/t/theoretical-limits-for-a-haproxy-instance/8168
Custom kernel parameters:
net.ipv4.ip_local_port_range = "12768 60999"
net.nf_conntrack_max = 5000000
fs.nr_open = 5000000
HAProxy config:
global
log /dev/log len 65535 local0 warning
chroot /var/lib/haproxy
stats socket /run/haproxy-admin.sock mode 660 level admin
user haproxy
group haproxy
daemon
maxconn 2000000
maxconnrate 2500
maxsslrate 2500
defaults
log global
option dontlognull
timeout connect 10s
timeout client 120s
timeout server 120s
frontend stats
mode http
bind *:8404
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /stats
stats refresh 10s
frontend k8s-api
bind *:6443
mode tcp
option tcplog
timeout client 300s
default_backend k8s-api
backend k8s-api
mode tcp
option tcp-check
timeout server 300s
balance leastconn
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s
maxconn 500 maxqueue 256 weight 100
server master01 x.x.x.x:6443 check
server master02 x.x.x.x:6443 check
server master03 x.x.x.x:6443 check
retries 0
frontend k8s-server
bind *:80
mode http
http-request add-header X-Forwarded-Proto http
http-request add-header X-Forwarded-Port 80
default_backend k8s-server
backend k8s-server
mode http
balance leastconn
option forwardfor
default-server inter 10s downinter 5s rise 2 fall 2 check
server worker01a x.x.x.x:31551 maxconn 200000
server worker02a x.x.x.x:31551 maxconn 200000
server worker03a x.x.x.x:31551 maxconn 200000
server worker04a x.x.x.x:31551 maxconn 200000
server worker05a x.x.x.x:31551 maxconn 200000
server worker06a x.x.x.x:31551 maxconn 200000
server worker07a x.x.x.x:31551 maxconn 200000
server worker08a x.x.x.x:31551 maxconn 200000
server worker09a x.x.x.x:31551 maxconn 200000
server worker10a x.x.x.x:31551 maxconn 200000
server worker11a x.x.x.x:31551 maxconn 200000
server worker12a x.x.x.x:31551 maxconn 200000
server worker13a x.x.x.x:31551 maxconn 200000
server worker14a x.x.x.x:31551 maxconn 200000
server worker15a x.x.x.x:31551 maxconn 200000
server worker16a x.x.x.x:31551 maxconn 200000
server worker17a x.x.x.x:31551 maxconn 200000
server worker18a x.x.x.x:31551 maxconn 200000
server worker19a x.x.x.x:31551 maxconn 200000
server worker20a x.x.x.x:31551 maxconn 200000
server worker01an x.x.x.x:31551 maxconn 200000
server worker02an x.x.x.x:31551 maxconn 200000
server worker03an x.x.x.x:31551 maxconn 200000
retries 0
frontend k8s-server-https
bind *:443 ssl crt /etc/haproxy/certs/
mode http
http-request add-header X-Forwarded-Proto https
http-request add-header X-Forwarded-Port 443
http-request del-header X-SERVER-SNI
http-request set-header X-SERVER-SNI %[ssl_fc_sni] if { ssl_fc_sni
-m found }
http-request set-var(txn.fc_sni) hdr(X-SERVER-SNI) if {
hdr(X-SERVER-SNI) -m found }
http-request del-header X-SERVER-SNI
default_backend k8s-server-https
backend k8s-server-https
mode http
balance leastconn
option forwardfor
default-server inter 10s downinter 5s rise 2 fall 2 check
no-check-ssl
server worker01a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker02a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker03a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker04a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker05a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker06a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker07a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker08a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker09a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker10a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker11a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker12a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker13a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker14a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker15a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker16a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker17a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker18a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker19a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker20a x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker01an x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker02an x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
server worker03an x.x.x.x:31445 ssl ca-file /etc/haproxy/ca/ca.crt
sni var(txn.fc_sni) maxconn 200000
retries 0
frontend k8s-nfs-monitor
bind *:8080
mode http
monitor-uri /health_nfs_cluster
acl k8s_server_down nbsrv(k8s-server) le 2
acl nfs_down nbsrv(nfs) lt 1
monitor fail if nfs_down || k8s_server_down
backend nfs
mode tcp
default-server inter 5s downinter 2s rise 1 fall 2
server nfs01 x.x.x.x:2049 check
frontend k8s-cluster-monitor
bind *:8081
mode http
monitor-uri /health_cluster
acl k8s_server_down nbsrv(k8s-server) le 2
monitor fail if k8s_server_down
Thanks in advance.
