Here are the 502 logs from the last run after the config changes.
2024-08-26T09:29:02.547581-06:00 testserver haproxy[284569]: <134>Aug 26
09:29:02 haproxy[284569]: 192.168.69.101:45382 [26/Aug/2024:09:29:02.545]
www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 5/5/3/3/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
2024-08-26T11:27:20.748921-06:00 testserver haproxy[284569]: <134>Aug 26
11:27:20 haproxy[284569]: 192.168.69.101:50606 [26/Aug/2024:11:27:20.746]
www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 5/5/3/3/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
2024-08-26T14:11:11.289987-06:00 testserver haproxy[284569]: <134>Aug 26
14:11:11 haproxy[284569]: 192.168.69.101:40516 [26/Aug/2024:14:11:11.285]
www~ front3/pdafront32 0/0/0/-1/2 502 208 - - SH-- 15/15/7/7/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
2024-08-26T17:40:55.801154-06:00 testserver haproxy[284569]: <134>Aug 26
17:40:55 haproxy[284569]: 192.168.69.101:53952 [26/Aug/2024:17:40:55.798]
www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 10/10/1/1/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
I'm open to suggestions.
Thanks,
BJ
On Fri, Aug 23, 2024 at 2:20 PM BJ Taylor <b...@greenseedtech.com> wrote:
> I will make the suggested changes and try again. Here are the failure logs
> for the last run. Queues are 0, current connections are not all that high.
>
> Aug 22 01:12:12 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:01:12:12
>> -0600","haproxy":{"connections":{"active":27,"frontend":27,"backend":13,"server":13},"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":47794,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
>> Aug 22 02:57:48 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:02:57:48
>> -0600","haproxy":{"connections":{"active":18,"frontend":18,"backend":3,
>> "server":3
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":51728,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":250,
>> "read":208}}}
>> Aug 22 03:22:34 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:03:22:34
>> -0600","haproxy":{"connections":{"active":14,"frontend":14,"backend":4,
>> "server":4
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":222,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":223},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":36912,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
>> Aug 21 16:26:12 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:16:26:12
>> -0600","haproxy":{"connections":{"active":8, "frontend":8, "backend":3,
>> "server":3
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":224,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":224},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":35664,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
>> Aug 21 20:43:51 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:20:43:51
>> -0600","haproxy":{"connections":{"active":28,"frontend":28,"backend":4,
>> "server":4
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":101,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":104},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.6.3",
>>
>> "client_port":39160,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/",
>> "protocol":"HTTP/2.0","header":{"host":"front1.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front1",
>> "server":"testbackend2"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65848,"read":208}}}
>> Aug 21 22:15:29 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:22:15:29
>> -0600","haproxy":{"connections":{"active":19,"frontend":19,"backend":4,
>> "server":4
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":221,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":222},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":34646,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
>> Aug 21 22:24:00 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:22:24:00
>> -0600","haproxy":{"connections":{"active":9, "frontend":9, "backend":4,
>> "server":4
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":269,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":269},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":44464,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":33018,"read":208}}}
>> Aug 21 22:41:33 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"21/Aug/2024:22:41:33
>> -0600","haproxy":{"connections":{"active":9, "frontend":9, "backend":4,
>> "server":4
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":291,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":292},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":36420,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>> "server":"testbackend1"},
>>
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":49402,"read":208}}}
>
>
> Thanks,
> BJ
>
>
> On Fri, Aug 23, 2024 at 12:50 PM Lukas Tribus <lu...@ltri.eu> wrote:
>
>> On Fri, 23 Aug 2024 at 18:55, BJ Taylor <b...@greenseedtech.com> wrote:
>> >
>> > We are trying to deploy HAProxy into our environment. We have a script
>> that
>> > does some 600k api calls during approximately 24 hours.
>>
>> How many concurrent connections / transactions though?
>>
>>
>> > During that time, when haproxy is in place, there are a handful (8-12)
>> of
>> > responses that come back as 502 with SH--.
>>
>> As per the documentation:
>>
>> SH The server aborted before sending its full HTTP response
>> headers, or
>> it crashed while processing the request. Since a server
>> aborting at
>> this moment is very rare, it would be wise to inspect its logs
>> to
>> control whether it crashed and why. The logged request may
>> indicate a
>> small set of faulty requests, demonstrating bugs in the
>> application.
>> Sometimes this might also be caused by an IDS killing the
>> connection
>> between HAProxy and the server.
>>
>>
>> You should disable custom format logging, enable httplog format and
>> share the affected SH log line.
>>
>>
>>
>> > tune.bufsize 8388608
>> > tune.maxrewrite 1024
>>
>> bufsize should usually be 16K maybe 32K for some specific application
>> requiring huge headers, but not 8M. I think 8M is unreasonable and I
>> think it will lead to issues one way or another.
>>
>>
>> > timeout connect 86400s
>> > timeout client 86400s
>> > timeout server 86400s
>>
>> 1 day long timeouts will probably also lead to an issue at some point,
>> due to sessions not expiring.
>>
>>
>>
>>
>> Lukas
>>
>
