Please note that ppa is built using USE_QUIC_OPENSSL_COMPAT=1 which is not fully QUIC, but a simulated QUIC on top of OpenSSL. it misses QUIC features like 0-RTT: SSL Libraries Support Status · haproxy/wiki Wiki <https://github.com/haproxy/wiki/wiki/SSL-Libraries-Support-Status>
OpenSSL-3.0 is known for degradation on highly concurrent systems, i.e. with many simultaneous threads running. especially in establishing new connections, maybe you won't observe it on long-running keepalive mode. ср, 8 янв. 2025 г. в 12:09, Lucas Rolff <lu...@lucasrolff.com>: > Hello, > > Sorry for my lengthy post, but I wanted to give as much info upfront as > possible, since it takes a bunch of guesswork out of it! > > I've recently started testing a combo of HAProxy 3.1 and Varnish 7.6 for > some content delivery / offloading, and I'm a bit curious if people have > any data/suggestions/optimizations that can be done to push things further > in terms of performance. > > I tried to use the HAProxy PPA for Ubuntu 24.04 (noble) ( > https://launchpad.net/~vbernat/+archive/ubuntu/haproxy-3.1 ), thanks > Vincent for providing these! > The build from the PPA uses the OS distributed OpenSSL, which is 3.0.13 on > Ubuntu 24.04. > I also have a custom build where I compiled in AWS-LC version 1.42. > > PPA distribution: > Build options : > TARGET = linux-glibc > CC = x86_64-linux-gnu-gcc > CFLAGS = -O2 -g -fwrapv -g -O2 -fno-omit-frame-pointer > -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects > -fstack-protector-strong -fstack-clash-protection -Wformat > -Werror=format-security -fcf-protection > -fdebug-prefix-map=/build/haproxy-ScKxv0/haproxy-3.1.1=/usr/src/haproxy-3.1.1-1ppa1~noble > -Wdate-time -D_FORTIFY_SOURCE=3 > OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_OT=1 USE_QUIC=1 > USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1 > DEBUG = > > Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY > +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE > -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH > -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC > -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL > -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT > +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP > +TPROXY -WURFL -ZLIB > > My own build: > Build options : > TARGET = linux-glibc > CC = cc > CFLAGS = -O2 -g -fwrapv > OPTIONS = USE_OPENSSL_AWSLC=1 USE_SLZ=1 USE_QUIC=1 USE_PROMEX=1 > USE_PCRE2=1 USE_PCRE2_JIT=1 > DEBUG = > > Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY > +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE > -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH > -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC > -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL > -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT > +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP > +TPROXY -WURFL -ZLIB > > I know there's quite a few differences in the CFlags, but I went with it > anyway! > > Test System: > - E5-2698v4 (20 cores, 40 threads) > - 128GB of 2133MHz DDR4 RAM (16GB DIMMs, in the correct banks) > - Ubuntu 24.04 with generic kernel > > Haproxy config: > global > log /dev/log local0 > log /dev/log local1 notice > chroot /var/lib/haproxy > stats socket /run/haproxy/admin.sock mode 660 level admin > stats timeout 30s > user haproxy > group haproxy > maxconn 50000 > daemon > > # Default SSL material locations > ca-base /etc/ssl/certs > crt-base /etc/ssl/private > > # See: > https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate > ssl-default-bind-ciphers > ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 > ssl-default-bind-ciphersuites > TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 > ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets > > defaults > log global > mode http > option httplog > option dontlognull > timeout connect 5000 > timeout client 50000 > timeout server 50000 > errorfile 400 /etc/haproxy/errors/400.http > errorfile 403 /etc/haproxy/errors/403.http > errorfile 408 /etc/haproxy/errors/408.http > errorfile 500 /etc/haproxy/errors/500.http > errorfile 502 /etc/haproxy/errors/502.http > errorfile 503 /etc/haproxy/errors/503.http > errorfile 504 /etc/haproxy/errors/504.http > > frontend ft_web > bind *:80,:::80 v6only > bind *:443,:::443 v6only ssl crt /etc/haproxy/ssl/ > bind quic4@:443 ssl crt /etc/haproxy/ssl/ alpn h3 > bind quic6@:443 ssl crt /etc/haproxy/ssl/ alpn h3 > > http-after-response add-header alt-svc 'h3=":443"; ma=60' > > option forwardfor > no log > > http-request set-header X-Forwarded-Proto https if { ssl_fc } > http-request set-header X-Forwarded-Proto http if !{ ssl_fc } > > default_backend bk_varnish > > backend bk_varnish > mode http > no log > server varnish1 127.0.0.1:8443 send-proxy > > > The SSL certificate being used is a P-256 ECC certificate. > > h2load using 14 threads, 100k requests, 100 clients and a 10 concurrent > streams per client, requesting a 2 megabyte jpeg image. > > It's worth noting with raw varnish (127.0.0.1:6081) I am able to push the > system to 175Gbps of traffic being served out of Varnish via h2c, this > obviously brings the challenge of having to run h2load on the same system, > since I.. sadly do not have 2x 200G connected servers! > > If I use haproxy for terminating SSL using OpenSSL 3.0, I am able to do > 63.76Gbps of traffic over http2 (h2) > Doing the same test using haproxy that uses AWS-LC v1.42, I get 64.21Gbps > of traffic over http2 (h2) > > Repeating the tests, averaging them out, it ends up being within margin of > error of each other. > > Question is, whether this is to be expected, that they'll perform roughly > the same in semi-large chunks of data (2MB), or if there's some key CFlags > options that I am missing to maybe push it a bit further > > From the various posts on the interwebs, and the haproxy wiki on GitHub, > it seems that people in general do not recommend OpenSSL 3.0 due to it's > bad performance, and AWS-LC should perform quite a bit better. > But is that largely for small files or should it be overall an improvement? > > Is there anything I could possibly do to push things further, target > filesize being 2 megabyte, which is served straight out of memory from > Varnish > > It's worth noting I did the same test on an EPYC 7502, and I get only > marginally better performance, at around 68Gbps of SSL traffic (180Gbps of > h2c via Varnish directly). > > I know 68Gbps also means doing that on localhost on top, due to the > backend bk_varnish. > > Another thing I noted, while testing QUIC/HTTP3 as well, is that haproxy > seems to consume about 4x as much resources to serve the same amount of > traffic (currently ~ 2Gbps of actual traffic) than it does for h2, is that > something that's likely to improve in the future? > > Best Regards, > Lucas Rolff > >
