Hi list,
we have a case where a response results into a 500 (PH):
Mar 1 14:44:05.000 n095211 haproxy[2427573]: 1.2.3.4:22917
[01/Mar/2023:14:44:05.108] genfrontend_somefoo_stage~
genbackend_49140-somefoo_stage/localhost 1/0/0/-1/533 500 20378 - - PH--
24/1/0/0/0 0/0
{stage.somedom|someUA|https://stage.somedom/customizing/index.xhtml?cid=4||https://stage.somedom||somcapture||id-ecPublicKey}
"POST /customizing/index.xhtml?cid=4 HTTP/1.1"
On the backend:
192.168.95.211 - - [01/Mar/2023:14:44:05 +0100] "POST
/customizing/index.xhtml?cid=4 HTTP/1.1" 200 27643
"https://stage.somedom/customizing/index.xhtml?cid=4"; "someUA"
So PH is stated as following:
PH The proxy blocked the server's response, because it was
invalid,
incomplete, dangerous (cache control), or matched a security
filter.
In any case, an HTTP 502 error is sent to the client. One
possible
cause for this error is an invalid syntax in an HTTP header
name
containing unauthorized characters. It is also possible but
quite
rare, that the proxy blocked a chunked-encoding request from
the
client due to an invalid syntax, before the server responded.
In this
case, an HTTP 400 error is sent to the client and reported in
the
logs. Finally, it may be due to an HTTP header rewrite failure
on the
response. In this case, an HTTP 500 error is sent (see
"tune.maxrewrite" and "http-response strict-mode" for more
inforomation).
In this case, the only relevant part seems to be:
Finally, it may be due to an HTTP header rewrite failure on the
response. In this case, an HTTP 500 error is sent (see
"tune.maxrewrite" and "http-response strict-mode" for more
inforomation).
tune.maxrewrite, hm, ok, it's set to 4096 in this case already. Also the
response header is really small and even below 1000 bytes so that
shouldn't be the problem, shouldn't it?
The response body is also pretty small compared to others but also the
body shouldn't matter in this case all all, shouldn't it? It's gzipped,
coming from an Apache, compressed it's between ~17-30kb, uncompressed
it's around 70-80kb.
Disabling strict mode is no option because we don't want to ignore
actual errors when processing / rewriting stuff internally in HAProxy.
I captured some of those responses but I don't see anything wrong or
weird at all to be honest. Another problem is, we cannot reproduce it
specifically. It only occurs sometimes, in some cases.
Any ideas?
HAProxy version 2.7.2-7e295dd 2023/01/20 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.2.html
Running on: Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21)
x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement
-Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2
-Wduplicated-cond -Wnull-dereference -fwrapv
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
-DMAX_SESS_STKCTR=12
OPTIONS = USE_PCRE= USE_PCRE_JIT= USE_PCRE2=1 USE_PCRE2_JIT=
USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS=
USE_SYSTEMD=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY
+CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE
+LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER
-NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2
-PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION
-QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO
+THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=4).
Built with OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto'
keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
--
Regards,
Christian Ruppert
