Hi, HAProxy 3.2-dev5 was released on 2025/02/08. It added 97 new commits after version 3.2-dev4.
The usual set of bug fixes (24 this time) got merged. One visible issue fixed just before this release was that the certificate selection could fail to pick an RSA-PSS certificate in TLSv1.3 when an RSA and an ECDSA certs are available for the same SNI. On the side of new stuff, the QUIC pacing was now marked as stable, as planned, which means that it's no longer needed to turn on the experimental directive to use it, it will be on by default (since it significantly reduces packet losses and is necessary for BBR anyway). It also means that BBR can be chosen without having to set the experimental directive. It is still possible to disable pacing, though (e.g. for troubleshooting). New termination events are available for logging. These include the sequence of state changes at various levels of the stack, and per side. The result is a sequence of letters and digits which can be decoded by dev/term_events/term_events. There are term_events at various levels in the stack (front/back connection, mux, stream, transaction etc) with new sample fetch functions for each of them, and an aggregate one called "term_events" which covers the whole chain at once. Once decoded, we can have a full sequence of events like this: $ dev/term_events/term_events f2x2f4x4 m2m4m1 e2e1 s2s1S1 E1 M1 F1 ### f2x2f4x4 : fd:shutr > xprt:shutr > fd:snd_err > xprt:snd_err ### m2m4m1 : muxc:shutr > muxc:snd_err > muxc:shutw ### e2e1 : se:eos > se:shutw ### s2s1S1 : strm:eos > strm:shutw > STRM:shutw ### E1 : SE:shutw ### M1 : MUXC:shutw ### F1 : FD:shutw It is way more detailed than the current termination codes, as it will permit to know precisely what side closed first, how, where errors were faced, etc. I think that initially this will mostly be requested by developers trying to distinguish a bug from an external wrong behavior, and might progressively be adopted by infrastructure operators who always have to justify why they think a problem doesn't come from the components they operate, just like with the term codes in the past ;-) On another point, we occasionally (maybe 1-2 times a year) get reports from users facing race conditions in epoll, which are extremely hard to troubleshoot. Usually they're caused by outdated or bogus kernels. We figured it was possible to detect them and even work around them. While some are unavoidable but harmless, others might (rarely) cause truncated connections by delivering a close event to an FD migrated twice and recycled in between. So we've added event counters for such cases, that can be observed in "debug counters | grep epoll", and these are now properly fixed by using an FD generation counter that allows to distinguish stale events from new ones. A few reports of heavily loaded machines (64 cores NUMA) where some shared tasks were causing unfairness in the scheduler and some threads could wait for a long time before getting the lock made us realize that we could improve the lock's fairness. The write lock would indeed always lose the race in front of an upgraded lock, which is not logical, so we changed this and this significantly reduced the worst case latency (typically 8 times lower, and 300 times less occurrences of latencies 32ms or above). Those who might have observed a few "traffic blocked" warnings could possibly see less (or even no more) of them, indicating improvements in the quality of service. On the quality of service front, it is now possible to switch to another task during TCP/HTTP rules evaluation in order to reduce latency. Some configs can have hundreds of expensive rules all based on regex for example, and evaluating all of them at once in a single call does cause important latencies. Now by default, evaluation will be interrupted every 50 rules, though this can be refined using tune.max-rules-at-once. Those seeking extremely low latencies could even go as low as 1 to evaluate a single rule at once and prevent complex rulesets from inflicting unacceptable latencies to other concurrent requests. It might for example make sense on shared LBs where some sites make a heavy use of rules and degrade the performance of other sites. Queues on large systems are known to be very CPU-intensive due to the cost of data sharing between CPU cores. The queues were thus refined to be thread-group aware. Each time a stream finishes, it will favor a pending request of the same group more often than those of other groups (1-to-10 by default). This is sufficient to totally eliminate this bottleneck, to the point where an extreme test on a 24-core EPYC used to trigger the watchdog in a few seconds, while now it performs linearly with the load thanks to the reduced sharing. Those using AMD CPUs with intense load variations should see significant improvements. The rest is mostly minor stuff like improved debugging output on some muxes and filters, as well as doc updates. Overall this version brings several improvements and as such comes with a bit more risks than the 4 previous ones. There's definitely a lot of reasons for adopting it, but you should watch closer "just in case". No worries, though, it's been running fine on haproxy.org over the last hour :-) Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/3.2/src/ Git repository : https://git.haproxy.org/git/haproxy.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy.git Changelog : https://www.haproxy.org/download/3.2/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Amaury Denoyelle (8): MINOR: quic: remove references to burst in quic-cc-algo parsing MINOR: quic: allow BBR testing without pacing MINOR: quic: transform pacing settings into a global option MAJOR: quic: mark pacing as stable and enable it by default MINOR: quic: mark BBR as stable MINOR: quic: define quic_tune BUILD: quic: fix overflow in global tune BUILD: quic: remove GCC undefined error in qc_release_lost_pkts() Aurelien DARRAGON (2): CLEANUP: tree-wide: define and use acl_match_cond() helper MEDIUM: stream: interrupt costly rulesets after too many evaluations Christopher Faulet (56): BUG/MINOR: stream: Properly handle "on-marked-up shutdown-backup-sessions" MEDIUM: stream: Map task wake up reasons to dedicated stream events MEDIUM: stream: No longer use TASK_F_UEVT* to shut a stream down MINOR: mux-h1: Add masks to group H1S DEMUX and MUX errors BUG/MINOR: mux-h1: Only report a SE error on demux error MINOR: tevt: Add the termination events log's fundations MINOR: tevt/stconn: Add a termination events log in the SE descriptor MINOR: tevt/mux-h1: Report termination events for the H1C and H1S MINOR: tevt/mux-h2: Report termination events for the H2C MINOR: tevt/stream/stconn: Report termination events for stream and sc MINOR: tevt/conn: Report intercepted event for L4 rules MINOR: tevt/mux-h1/mux-h2: Add termination events log when dumping mux info MINOR: tevt/muxes: Add CTL and SCTL command to get the termination event logs MINOR: tevt/mux-pt: Add support for termination event logs MINOR: tevt/connection: Add dedicated termination events for lower locations MEDIUM: tevt/muxes: Add dedicated termination events for muxc/se locations MINOR: tevt/stconn: Be more accurate to report shutw events MEDIUM: tevt/stconn/stream: Add dedicated termination events for stream location MINOR: tevt: Don't duplicate termination event during reporting MINOR: tevt/applet: Add limited support for termination event logs for applets MINOR: tevt: Add a sample to get termination events for all locations MINOR: tevt: Improve function to convert a termination events log to string REORG: tevt/connection: Move enums at the end of the header file MINOR: tevt/dev: Add term_events tool MINOR: tevt/connection: Add support for POLL_HUP/POLL_ERR events MINOR: tevt/dev: Parse tuple of termination events BUG/MEDIUM: mux-fcgi: Properly handle read0 on partial records BUG/MINOR: tevt/http-ana: Remove badly placed event reports DEBUG: http-ana: Remove debug counters from HTTP analyzers DEBUG: mux-h1: Remove some debug counters BUG/MINOR: tcp-rules: Don't forward close during tcp-response content rules eval BUG/MINOR: http-check: Don't pretend a C-L heeader is set before adding it BUG/MINOR: tevt/mux-h2: Set truncated receive/eos events at SE level on error BUG/MEDIUM: flt-spoe: Set/test applet flags instead of SE flags from I/O handler BUG/MEDIUM: applet: Don't pretend to have more data to handle EOI/EOS/ERROR BUG/MEDIUM: flt-spoe: Properly handle end of stream from the SPOE applet MINOR: flt-spoe: Report end of input immediately after applet init MINOR: mux-spop: Report EOI on the SE when a ACK is received for a stream MINOR: mux-spop: Set SPOP_CF_ERROR flag on connection error only MINOR: tevt/mux-spop: Report termination events for the SPOP connect/stream CLEANUP: mux-spop: Remove useless comments MINOR: mux-spop: Dump info about connections and streams in dedicated functions MINOR: mux-spop: Implement .show_sd callback function MEDIUM: mux-fcgi: Add a function to propagate termination flags from fstrm to SE BUG/MEDIUM: mux-fcgi: Propagate flags to SE in fcgi_strm_wake_one_stream MINOR: tevt/mux-fcgi: Report termination events for the FCGI connect/stream MINOR: mux-fcgi: Dump info about connections and streams in dedicated functions MINOR: mux-spop/mux-fcgi: Add support of the debug string for logs BUG/MINOR: cli: Don't set SE flags from the cli applet BUG/MINOR: cli: Fix memory leak on error for _getsocks command BUG/MINOR: cli: Fix a possible infinite loop in _getsocks() BUG/MINOR: config/userlist: Support one 'users' option for 'group' directive BUG/MINOR: auth: Fix a leak on error path when parsing user's groups BUG/MINOR: flt-trace: Support only one name option MINOR: filters: Improve errors formating during filters parsing BUG/MINOR: stats-json: Define JSON_INT_MAX as a signed integer Frederic Lecaille (1): BUILD: ssl: remove a boringssl definition defined by recent boringssl libs Lukas Tribus (1): DOC: option redispatch should mention persist options Olivier Houchard (6): MINOR: proxies: Add a per-thread group field to struct proxy. MINOR: Add fields to the per-thread group field in struct server. MINOR: proxies/servers: Calculate queueslength and use it. MEDIUM: servers/proxies: Switch to using per-tgroup queues. MINOR: queues: use __ha_cpu_relax() on failed CAS. BUILD: queues: Use unsigned int when needed Valentine Krasnobaeva (3): BUG/MINOR: ssl: put ssl_sock_load_ca under SSL_NO_GENERATE_CERTIFICATES CLEANUP: ssl: rename ssl_sock_load_ca to ssl_sock_gencert_load_ca CLEANUP: ssl: move ssl_sock_gencert_load_ca declaration in ssl_gencert.h William Lallemand (6): BUILD: ssl: allow to build without the renegotiation API of WolfSSL BUILD: ssl: more cleaner approach to WolfSSL without renegotiation BUG/MEDIUM: htx: wrong count computation in htx_xfer_blks() DOC: htx: clarify <mark> parameter for htx_xfer_blks() MEDIUM: htx: prevent <mark> to copy incomplete headers in htx_xfer_blks() BUG/MEDIUM: ssl: chosing correct certificate using RSA-PSS with TLSv1.3 Willy Tarreau (14): MINOR: epoll: permit to mask certain specific events BUILD: tools: fix build on BSD by dropping the ETIME check BUG/MEDIUM: chunk: make sure to flush the trash pool before resizing DEBUG: fd: add a counter of takeovers of an FD since it was last opened MINOR: fd: add a generation number to file descriptors DEBUG: epoll: store and compare the FD's generation count with reported event MEDIUM: epoll: skip reports of stale file descriptors BUG/MINOR: debug: make "debug dev sched" accept a negative TID BUG/MINOR: debug: make sure the "debug dev sched" tasks don't block stopping IMPORT: plock: export the uninlined version of the lock wait function IMPORT: plock: give higher precedence to W than S IMPORT: plock: lower the slope of the exponential back-off IMPORT: plock: use cpu_relax() for a shorter time in EBO Revert "IMPORT: plock: export the uninlined version of the lock wait function" ---
