I will make the suggested changes and try again. Here are the failure logs
for the last run. Queues are 0, current connections are not all that high.
Aug 22 01:12:12 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:01:12:12
> -0600","haproxy":{"connections":{"active":27,"frontend":27,"backend":13,"server":13},"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":47794,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
> Aug 22 02:57:48 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:02:57:48
> -0600","haproxy":{"connections":{"active":18,"frontend":18,"backend":3,
> "server":3
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":51728,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":250,
> "read":208}}}
> Aug 22 03:22:34 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:03:22:34
> -0600","haproxy":{"connections":{"active":14,"frontend":14,"backend":4,
> "server":4
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":222,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":223},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":36912,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
> Aug 21 16:26:12 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:16:26:12
> -0600","haproxy":{"connections":{"active":8, "frontend":8, "backend":3,
> "server":3
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":224,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":224},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":35664,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
> Aug 21 20:43:51 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:20:43:51
> -0600","haproxy":{"connections":{"active":28,"frontend":28,"backend":4,
> "server":4
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":101,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":104},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.6.3",
>
> "client_port":39160,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/",
> "protocol":"HTTP/2.0","header":{"host":"front1.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front1",
> "server":"testbackend2"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65848,"read":208}}}
> Aug 21 22:15:29 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:22:15:29
> -0600","haproxy":{"connections":{"active":19,"frontend":19,"backend":4,
> "server":4
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":221,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":222},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":34646,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
> Aug 21 22:24:00 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:22:24:00
> -0600","haproxy":{"connections":{"active":9, "frontend":9, "backend":4,
> "server":4
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":269,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":269},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":44464,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":33018,"read":208}}}
> Aug 21 22:41:33 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:22:41:33
> -0600","haproxy":{"connections":{"active":9, "frontend":9, "backend":4,
> "server":4
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":291,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":292},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":36420,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
> "server":"testbackend1"},
>
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":49402,"read":208}}}
Thanks,
BJ
On Fri, Aug 23, 2024 at 12:50 PM Lukas Tribus <lu...@ltri.eu> wrote:
> On Fri, 23 Aug 2024 at 18:55, BJ Taylor <b...@greenseedtech.com> wrote:
> >
> > We are trying to deploy HAProxy into our environment. We have a script
> that
> > does some 600k api calls during approximately 24 hours.
>
> How many concurrent connections / transactions though?
>
>
> > During that time, when haproxy is in place, there are a handful (8-12)
> of
> > responses that come back as 502 with SH--.
>
> As per the documentation:
>
> SH The server aborted before sending its full HTTP response
> headers, or
> it crashed while processing the request. Since a server aborting
> at
> this moment is very rare, it would be wise to inspect its logs to
> control whether it crashed and why. The logged request may
> indicate a
> small set of faulty requests, demonstrating bugs in the
> application.
> Sometimes this might also be caused by an IDS killing the
> connection
> between HAProxy and the server.
>
>
> You should disable custom format logging, enable httplog format and
> share the affected SH log line.
>
>
>
> > tune.bufsize 8388608
> > tune.maxrewrite 1024
>
> bufsize should usually be 16K maybe 32K for some specific application
> requiring huge headers, but not 8M. I think 8M is unreasonable and I
> think it will lead to issues one way or another.
>
>
> > timeout connect 86400s
> > timeout client 86400s
> > timeout server 86400s
>
> 1 day long timeouts will probably also lead to an issue at some point,
> due to sessions not expiring.
>
>
>
>
> Lukas
>
