Hi, HAProxy 3.1-dev11 was released on 2024/11/01. It added 125 new commits after version 3.1-dev10.
We're stabilizing. 37 bugs fixed in one week. I'm not going to review them, that's far too long and inaccurate, and it'd remind bad memories. Still some fixes to come, but the team is hard at work on this so I'm confident. A number of the long painful ones pre-dating 3.0 or even 2.9 were fixed. A big thanks by the way to Felipe and Ricardo for the energy they've been dedicating to chase a difficult one. I guess we've inflated their logs quite a bit :-) Speaking of this, we've also improved the debugging capabilities, in order to permit to place arbitrary event counters anywhere in the code and check them from the CLI at run time. Dealing with truncated core dumps and some without symbols also made us realize that we were not fully leveraging the power of the post_mortem struct, that has been significantly enhanced to help navigate through a core even without symbols and figure many internal entities to try to figure what was happening. In parallel, we're often missing some stderr outputs of the backtraces, and the core only contains the last one due to the same buffer being used in turn with each thread. Now each thread now dumps its own backtrace in its own buffer, so that the core will contain them all. Finally, some gdb scripts that we're using for inspecting cores were merged (see dev/gdb). Hopefully future bugs will take less guess work. Some of this work was backported to 3.0 and 2.9 so that we improve our responsiveness and accuracy on these versions as well. A pair of new sample fetch functions was added, "last_entity" and "waiting_entity". They indicate what waiting operation was interrupted by a timeout or an error, which can often indicate that such error is in fact in the designated entity. For example if you write a Lua filter, if there's a bug in an SPOA or a bug in the compression and a timeout strikes, it will be possible to know that the filter was still waiting for something that did not happen. Waiting for a body is also among the detected causes (and reported as the backend or frontend depending where it happens). Speaking about debugging for logs, the H1 mux now feeds the debug_str functions with its internal states. A new converter "when()" will only let the input sample pass through when a condition is met. For now conditions are very scarce (error, stopping and a few other ones). The goal is to permit to emit some complementary info in logs only when really needed (such as the debug_str output). The abstract socket variant started by Tristan 8 months ago was finally rebased and merged with the help of Aurélien. This means that now there's a new family "abnsz" which indicates a zero-terminated ABNS address that is compatible with certain tools that use them this way, so now we're compatible with both practices. That's true on both sides, both front and back. Thanks guys for finishing that work! Small but useful stuff, the CLI now learned a new "echo" command, that does the same as in shell. The purpose is to help placing delimiters in dumps made of the output of multiple chained commands. I used to rely on short commands like "show version" to place ugly delimiters in diagnostic scripts, naturally "echo" will be more convenient and will also permit to show the command that is run for example. Two new converters, rfc7239_nn and rfc7239_np, permit one to build their own custom Forwarded header based on any input data from the http-request ruleset. QUIC now offers a new token on every new connection, not just 0-RTT ones, so that the client can reuse it later, possibly avoiding one RTT for address validation. New CI updates regarding LibreSSL, MacOS updated to 15 and the entry of Ubuntu 24.04. The rest is cleanups (particularly in the master-worker area), a bit more traces, doc updates, and a bunch of everything as usual. Again, please test it. It's much better than dev10 in terms of stability, and it helps us fix trouble before it gets really painful. We're still aware of some pending bug reports but they also affect 3.0 and sometimes even 2.9, so if you're "lucky" enough to witness them, the amount of debugging facilities in 3.1 is vastly superior to previous versions and should help us figure faster what is happening. Similarly if you're currently facing bugs on 3.0.5, please give a try to 3.1-dev11. Some issues are already gone and the other ones will be easier to debug there. We plan to issue 3.0.6 shortly anyway but we'd prefer it with the least possible known pending bugs. I plan to issue 3.1-dev12 next week, but I'll be at the HTTP Workshop so it will likely be delated to Friday (unless someone else does it before). Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/3.1/src/ Git repository : https://git.haproxy.org/git/haproxy.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy.git Changelog : https://www.haproxy.org/download/3.1/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Amaury Denoyelle (6): MINOR: mux-quic: simplify sending of empty STREAM FIN BUG/MINOR: mux-quic: do not close STREAM with empty FIN if no data sent BUG/MINOR: server: fix dynamic server leak with check on failed init BUG/MEDIUM: server: fix race on servers_list during server deletion MINOR: quic: complete trace in qc_may_build_pkt() MINOR: quic: move qc_send_mux() prototype into quic_tx.h Aurelien DARRAGON (13): DOC: config: fix rfc7239 forwarded typo in desc MINOR: http_ext: implement rfc7239_{nn,np} converters CLEANUP: http_ext: remove useless BUG_ON() in http_handle_xot_header() BUG/MEDIUM: connection/http-reuse: fix address collision on unhandled address families MINOR: tools: add strnlen2() helper CLEANUP: log: use strnlen2() in _lf_text_len() to compute string length MEDIUM: protocol: rely on AF_CUST_ABNS family to recognize ABNS sockets CLEANUP: tools: rely on address family to detect ABNS sockets MINOR: sock: restore effective UNIX family in sock_get_old_sockets() MEDIUM: sock: also restore effective unix family in get_{src,dst}() MEDIUM: sock_unix: use per-family addrcmp function DOC: config: add missing glitch_{cnt,rate} data types DOC: config: add missing glitch_{cnt,rate} sample definitions Christopher Faulet (30): BUG/MEDIUM: stconn: Wait iobuf is empty to shut SE down during a check send MINOR: mux-h1: Show the SD iobuf in trace messages on stream send events MINOR: mux-h1: Add a trace on shutdown when keep-alive is not possible BUG/MINOR: http-ana: Don't report a server abort if response payload is invalid BUG/MEDIUM: stconn: Check FF data of SC to perform a shutdown in sc_notify() BUG/MAJOR: filters/htx: Add a flag to state the payload is altered by a filter REGTESTS: Never reuse server connection in http-messaging/truncated.vtc BUG/MEDIUM: stats-html: Never dump more data than expected during 0-copy FF BUG/MEDIUM: mux-h2: Remove H2S from send list if data are sent via 0-copy FF BUG/MINOR: stconn: Pretend the SE have more data to deliver on abortonclose CLEANUP: stream: remove outdated comments DEBUG: stream: Add debug counters to track some client/server aborts DEBUG: mux-h1: Add debug counters to track some errors MINOR: mux-h1: Add support of the debug string for logs DEBUG: mux-h1: Add debug counters to track errors with in/out pending data BUG/MINOR: mux-h1: Fix conditions on pipe in some COUNT_IF() BUG/MEDIUM: stconn: Report blocked send if sends are blocked by an error BUG/MINOR: http-ana: Fix wrong client abort reports during responses forwarding BUG/MINOR: stconn: Don't disable 0-copy FF if EOS was reported on consumer side DEBUG: mux-h1: Add H1C expiration dates in trace messages BUG/MEDIUM: mux-h1: Fix how timeouts are applied on H1 connections BUG/MINOR: http-ana: Report internal error if an action yields on a final eval MINOR: stream: Save last evaluated rule on invalid yield MINOR: stream: Replace last_rule_file/line fields by a more generic field MINOR: stream: Save the last filter evaluated interrupting the processing MINOR: stream: Save the entity waiting to continue its processing MINOR: stream: Use an enum to identify last and waiting entities for streams MINOR: stream: Add http-buffer-request option in the waiting entities DOC: config: Add documentation about last_entity sample fetch DOC: config: Add documentation about waiting_entity sample fetch Dragan Dosen (2): BUG/MINOR: sample: free err2 in smp_resolve_args for type ARGT_REG MINOR: arg: add an argument type for identifier Frederic Lecaille (3): BUG/MINOR: quic: avoid leaking post handshake frames MINOR: quic: send new tokens (NEW_TOKEN) even for 1RTT sessions BUG/MEDIUM: quic: avoid freezing 0RTT connections Ilia Shipitsin (4): CI: switch QUIC Interop on AWS-LC to common docker image CI: switch QUIC Interop on LibreSSL to common docker image CI: enable chacha20 test on LibreSSL QUIC Interop CI: LibreSSL QUIC Interop: fix docker context Ilya Shipitsin (3): CI: prepare Coverity build for Ubuntu 24 CI: bump development builds explicitely to Ubuntu 24.04 CI: modernize macos builds to macos-15 Tristan (1): MEDIUM: socket: add zero-terminated ABNS alternative Valentine Krasnobaeva (17): BUG/MINOR: mworker: fix mworker-max-reloads parser BUG/MINOR: mworker/cli: show master startup logs in recovery mode MINOR: mworker: stop MASTER proxy listener on worker mcli sockpair MINOR: error: simplify startup_logs_init_shm BUG/MINOR: mworker: show worker warnings in startup logs CLEANUP: mworker: clean mworker_reexec MINOR: mworker/cli: split mworker_cli_proxy_create CLEANUP: mworker: make mworker_create_master_cli more readable BUG/MEIDUM: mworker: fix fd leak from master to worker BUG/MINOR: mworker/cli: fix mworker_cli_global_proxy_new_listener BUG/MINOR: mworker: mworker_reexec: unset MODE_STARTING before free startup logs ring BUG/MINOR: errors: startup_logs_free: set global startup_logs ptr to NULL BUG/MINOR: errors: print_message: don't allocate startup logs ring BUG/MINOR: startup: don't fork worker if started with -c -W BUG/MINOR: startup: dump libs only in worker if started with -W -dL BUG/MINOR: startup: dump keywords only in worker if started with -W -dKAll BUG/MINOR: startup: don't dump polling info for master in verbose mode William Lallemand (11): BUG/MINOR: httpclient: return NULL when no proxy available during httpclient_new() BUG/MEDIUM: mworker/httpclient: initialization skipped by accident in mworker mode BUG/MINOR: resolvers/mworker: missing default resolvers in mworker mode MINOR: mworker/ocsp: skip ocsp-update proxy init in master MINOR: mworker/cli: add 'debug' to 'show proc' MINOR: mworker/cli: remove comment line for program when useless MINOR: mworker/cli: 'show proc debug' for old workers MINOR: cli: remove non-printable characters from 'debug dev fd' MINOR: cli: add an 'echo' command BUG/MINOR: trace: stop rewriting argv with -dt BUG/MINOR: ssl/cli: 'set ssl cert' does not check the transaction name correctly Willy Tarreau (35): BUILD: buffers: keep b_getblk_nc() and b_peek_varint() in buf.h CLEANUP: buffers: simplify b_get_varint() OPTIM: buffers: avoid a useless wrapping check for ofs == 0 MINOR: debug: make mark_tainted() return the previous value MINOR: chunk: drop the global thread_dump_buffer MINOR: debug: split ha_thread_dump() in two parts MINOR: debug: slightly change the thread_dump_pointer signification MINOR: debug: make ha_thread_dump_done() take the pointer to be used MINOR: debug: replace ha_thread_dump() with its two components MEDIUM: debug: on panic, make the target thread automatically allocate its buf BUILD: mux-h2/traces: fix build on 32-bit due to size of the DATA frame CLEANUP: debug: make the BUG_ON() macros check the condition in the outer one MEDIUM: debug: add match counters for BUG_ON/WARN_ON/CHECK_IF MINOR: debug: add a new debug macro COUNT_IF() MINOR: debug: add "debug dev counters" to list code counters MINOR: stream: maintain per-stream counters of the number of passes on code MINOR: filters: add per-filter call counters MINOR: sample: add the "when" converter to condition some expressions BUILD: spoe: fix build warning on older gcc around sub-struct initialization Revert "OPTIM: mux-h2: make h2_send() report more accurate wake up conditions" MINOR: activity/memprofile: show per-DSO stats BUILD: debug: silence a build warning with threads disabled CLEANUP: mux-h2: remove the unused "full" variable in h2_frt_transfer_data() MINOR: pools: export the pools variable MINOR: debug: place a magic pattern at the beginning of post_mortem MINOR: debug: place the post_mortem struct in its own section. MINOR: debug: store important pointers in post_mortem MINOR: debug: do not limit backtraces to stuck threads MINOR: debug: also add a pointer to struct global to post_mortem DOC: design: add notes about more detailed error reporting for logs MINOR: debug: also add fdtab and acitvity to struct post_mortem MINOR: debug: remove the redundant process.thread_info array from post_mortem DEV: gdb: add a number of gdb scripts to navigate in core dumps MEDIUM: protocol: make abns a custom unix socket address family MINOR: protocol: create abnsz socket address family ---
