Hello Marc,
one another:
source ipv4@ usesrc clientip
hope that helps.
Stefan
Am 17.02.23 um 12:47 schrieb Marc West:
Hi,
After my other thread about performance issues on OpenBSD we decided to
switch OSes on our HAProxy boxes to FreeBSD 13.1. In the test
environment everything worked perfectly with transparent proxying but
when cutting production over to FreeBSD I ran into an issue and had to
revert for now.
With "source 0.0.0.0 usesrc client" everything seemed OK for about a
minute but then many client connections across all backends started
getting 503 errors (or connection failures for TCP backends). The
haproxy logs have a lot of:
- Cannot bind to tproxy source address before connect() for backend my_backend
- Connect() failed for backend my_backend: local address already in use.
This was during a late night maintenance with less than 100 connections,
netstat had less than 500 total connections, and I didn't see any limits
that were close to being hit (these are all auto-detected except
nbthread=36 to leave some breathing room on this 40 core HW):
pid = 76927 (process #1, nbproc = 1, nbthread = 36)
uptime = 0d 0h14m15s
system limits: memmax = unlimited; ulimit-n = 940059
maxsock = 940059; maxconn = 469926; maxpipes = 0
current conns = 40; current pipes = 0/0; conn rate = 8/sec; bit rate = 1.086
Mbps
Running tasks: 0/248; idle = 100 %
Health checks were always OK and the 503s/failures were intermittent
with a majority but not all requests failing. With "clientip" instead of
"client" it seemed to be worse with almost all requests failing.
Looking back at the test env there were some occurrences of the
"Connect() failed ... address already in use" in logs but there were no
noticeable request failures. There were zero "Cannot bind to tproxy
source address before connect()" in test even with much higher stress
testing load. Configs between test and production are nearly identical
except there are more backends and VLANs in production and prod traffic
comes from more IPs than the 2-3 IPs that hit test.
On OpenBSD transparent works out of the box with no PF divert rules
needed (only standard pass rules). Unfortunately with FreeBSD's PF
(which is behind OpenBSD on features but ahead on performance) I have
not found any combination of rules that allow transparent to work at
all (and divert-reply isn't implemented), so here ipfw is diverting and
passing all traffic stateless like so:
$cmd 0001 fwd localhost tcp from 10.10.15.0/24 80,443 to any in recv vlan123
$cmd 65534 allow all from any to any
Then PF handles the main stateful firewall rules and outbound NAT for
internal servers. All PF rules have logging and there was nothing being
blocked by PF during the issue.
It isn't ideal to run both ipfw and PF like this but it would be a
major undertaking to rework the tooling we have built around PF to a
different firewall, and since transparent is required it seems ipfw is
needed to handle the divert part on FreeBSD.
Am I overlooking something in my setup or might this be an OS or haproxy
issue to debug further? Thanks in advance for any thoughts!
$ haproxy -vv
HAProxy version 2.6.9-3a3700a 2023/02/14 -https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs:http://www.haproxy.org/bugs/bugs-2.6.9.html
Running on: FreeBSD 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -Wall
-Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
-DFREEBSD_PORTS
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1
USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 -BACKTRACE +CLOSEFROM +CPU_AFFINITY -CRYPT_H
-DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE +LIBCRYPT
-LINUX_SPLICE -LINUX_TPROXY +LUA -MEMORY_PROFILING -NETFILTER -NS
-OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL
+PROCCTL +PROMEX -QUIC -RT -SLZ -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO
+THREAD -THREAD_DUMP +TPROXY -WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=40).
Built with OpenSSL version : OpenSSL 1.1.1o-freebsd 3 May 2022
Running on OpenSSL version : OpenSSL 1.1.1o-freebsd 3 May 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 13.0.0 (g...@github.com:llvm/llvm-project.git
llvmorg-13.0.0-0-gd7b669b3a303)
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
$
