Hi Olivier.
On 2025-07-09 (Mi.) 12:44, Olivier D wrote:
Hello everyone,
I'm currently troubleshooting intermittent 520 errors returned by Cloudflare in
front of an HAProxy-based load balancing infrastructure.
These errors are rare—affecting about 0.001% of requests—which makes them
especially tricky to diagnose. According to Cloudflare, a 520 indicates the
origin server (HAProxy, in this case) unexpectedly closed the connection or
returned an empty response.
Key Observations
- it happens on several frontends and on different haproxy instances.
- The issue only occurs when the client request protocol is HTTP/2 (http/3 is
disabled on our side at the moment).
- Disabling Cloudflare's "HTTP/2 to origin" option completely eliminates the
issue.
This strongly suggests the problem lies in the interaction between Cloudflare's
HTTP/2 implementation and HAProxy's HTTP/2 support.
Cloudflare Logs for 520 Events
"OriginSSLProtocol": "unknown",
"OriginTCPHandshakeDurationMs": 11,
"OriginTLSHandshakeDurationMs": 12,
"OriginRequestHeaderSendDurationMs": 0,
"OriginResponseDurationMs": 0,
"ClientSSLCipher": "AEAD-AES128-GCM-SHA256",
"ClientSSLProtocol": "TLSv1.3"
To me, this implies:
- Cloudflare successfully opens a TCP connection to HAProxy
(OriginTCPHandshakeDurationMs is non-zero),
- But fails during the TLS handshake, as evidenced by "OriginSSLProtocol":
"unknown".
HAProxy version 2.8.15 (issue also present on 2.8.11)
Relevant global settings:
ssl-default-bind-ciphersuites
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-client-sigalgs
rsa_pkcs1_sha256:rsa_pkcs1_sha384:rsa_pkcs1_sha512:ecdsa_secp256r1_sha256:ecdsa_secp384r1_sha384:ecdsa_secp521r1_sha512:rsa_pss_rsae_sha256:rsa_pss_rsae_sha384:rsa_pss_rsae_sha512:ed25519:ed448:rsa_pss_pss_sha256:rsa_pss_pss_sha384:rsa_pss_pss_sha512
option tcp-smart-accept
option tcp-smart-connect
Frontend configuration (abridged):
mode http
bind xxxxx:443 ssl crt-list /folder/redacted.txt ssl-min-ver TLSv1.2 tls-
ticket-keys /var/run/tls_ticket_keys
I've tried enabling debug-style logging using:
option log-separate-errors
option logasap
error-log-format '%ci:%cp -> %fi:%fp %b/%s | Tq=%Tq Tw=%Tw Tc=%Tc Tr=%Tr
Ta=%Ta Tt=%Tt TR=%TR HTTP=%ST (%hrl) B=%B | TLS=%sslv/%sslc SNI=%[ssl_fc_sni]
%HM %HP'
Unfortunately, these logs don't show any detail about the failed TLS handshakes.
Questions
- have anyone experienced this issue ? If you have a small amount of 520 it may
not been printed on the interface, but it is there. Just go the following URL :
https://dash.cloudflare.com/[accountID]/mydomain.com/analytics/traffic?status-
code=520 <https://dash.cloudflare.com/[accountID]/mydomain.com/analytics/
traffic?status-code=520>
- Are there any known issues with HAProxy and HTTP/2 in TLS termination
scenarios ?
- Is there any way to increase logging detail specifically for failed TLS
handshakes? tcpdump is a no go in this case due to the heavy trafic and the
minimal occurrence of the issue.
Because the issue is so rare (under 100 requests/day), every test takes time to
yield results. Any insights or ideas would be greatly appreciated.
You can try to increase the `tune.maxrewrite` to `2048` and see if the issue is
fixed.
Thanks in advance!
Olivier
Regards
Aleks
