Also worth noting that Nginx OS does not experience the issues (otherwise same setup)...
Dan On Mon, Dec 20, 2021 at 10:58 AM Dan Washusen <d...@reactive.org> wrote: > Yeah, all those parameters are configured and everything works fine when I > switch to the MPM worker module. The 408 errors are returned by Jetty and > ProxyPass is configured to not reuse connections (which seems to suggest > not a KeepAlive issue), Jetty is erroring because it receives no request > data. Kind of seems like the MPM event module is missing events/data (when > behind the ALB)... > > Given that the TCP network load balancer doesn't show the issue, maybe > it's something to do with how the ALB sends the data to the instance (e.g. > Jumbo frames vs the standard MTU)...? > > On Mon, Dec 20, 2021 at 10:37 AM Igor Cicimov <icici...@gmail.com> wrote: > >> I will still say this is a timeout issue. Set the ALB timeout to 3600 >> which is the max possible in case you do not know where to start from. I >> guess you already have ALB logs enabled. >> >> In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout >> and make sure they make sense for your user case. Enable debug logs too for >> more details. >> >> It is tough to guess without knowing your relevant configuration but 408 >> is usually caused by client connection being closed while keep-alive in use. >> >> On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen <d...@reactive.org> wrote: >> >>> Thanks for the response. Timeouts are configured appropriately... >>> >>> To clarify; everything works fine through a TCP Network Load Balancer >>> pointing at the same infrastructure. There is something about having a HTTP >>> based Application Load Balancer in front of an MPM event configuration >>> that's causing issues... >>> >>> Dan >>> >>> >>> >>> On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov <icici...@gmail.com> >>> wrote: >>> >>>> In a proxy chains like this getting the timeouts in sync is the most >>>> important thing. Make sure that you have done that. >>>> >>>> On Mon, 20 Dec 2021, 08:37 Dan Washusen, <d...@reactive.org> wrote: >>>> >>>>> Hi All, >>>>> I've been experimenting with the MPM event module with Apache >>>>> instances sitting behind an AWS Application Load Balancer (ALB) and it >>>>> really doesn't seem to be working well. Response times shoot up (compared >>>>> to MPM event worker) and we see a fair few 502 errors returned (by the AWS >>>>> ALB. >>>>> >>>>> The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> >>>>> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers >>>>> >>>>> Debugging the issue I think I traced it down to Jetty returning a 408 >>>>> error because it can't read the request body in a timely manner. So it >>>>> seems like for some reason MPM is sending the request body...? >>>>> >>>>> We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the >>>>> following worker configuration: >>>>> >>>>> ServerLimit 250 >>>>> StartServers 100 >>>>> MinSpareThreads 75 >>>>> MaxSpareThreads 250 >>>>> ThreadLimit 64 >>>>> ThreadsPerChild 64 >>>>> MaxRequestWorkers 8000 >>>>> >>>>> I've come across several random posts mentioning that the MPM event >>>>> module doesn't work behind an ALB but no-one seems to be into any detail. >>>>> Anyone have some debugging/configuration suggestions? >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>> p.s. I've created a serverfault post showing graphs etc: >>>>> https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >>>>> >>>>
