On Thu, Mar 13, 2025 at 5:01 AM Tobia Scapin <to...@scapin.eu> wrote: > > Dear apache users, > I'm looking for help with MPM events settings and scoreboard full error. > I read many suggestions but not really a guide to understand what happened. > > I have a webserver that mainly is used for php pages with fpm and as > websocket proxy. > > These are some details or Apache installation: > Ubuntu 20.04.6 LTS - 8GB RAM 2 cpu > Server Version: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.1.1f > Server MPM: event > Server Built: 2024-07-17T18:58:09 > > Yesterday I started to get this error: > [Wed Mar 12 21:48:26.198131 2025] [mpm_event:error] [pid 3510776:tid > 139759223077952] AH03490: scoreboard is full, not at > MaxRequestWorkers.Increase ServerLimit. > > At midnight after logrotation restart, the webserver became completely > not reachable and it made just the same "scoreboard is full" log every > second. > I manually restarted apache service this morning and then everything > came back to normal. During shutdown there were some logs like this " > AH00046: child process 178881 still did not exit, sending a SIGKILL". > > Fortunately I had a cron to save apache status, so this is the status > at midnight after apache was completely unreachable (and it stopped > event to reply to status request). And this is the output: > > Apache Server Status for localhost (via 127.0.0.1) > > Server Version: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.1.1f > Server MPM: event > Server Built: 2024-07-17T18:58:09 > > ------------------------------------------------------------------------------- > > Current Time: Thursday, 13-Mar-2025 00:00:01 CET > Restart Time: Thursday, 06-Mar-2025 11:27:01 CET > Parent Server Config. Generation: 8 > Parent Server MPM Generation: 7 > Server uptime: 6 days 12 hours 32 minutes 59 seconds > Server load: 0.09 0.11 0.09 > Total accesses: 597286 - Total Traffic: 10.1 GB - Total Duration: 26001983413 > CPU Usage: u817.39 s221.47 cu948.69 cs297.75 - .405% CPU load > 1.06 requests/sec - 18.9 kB/second - 17.8 kB/request - 43533.6 ms/request > 20 requests currently being processed, 30 idle workers > > Slot PID Stopping Connections Threads Async connections > total accepting busy idle writing keep-alive > closing > 0 3510778 yes (old gen) 4 no 0 0 0 0 0 > 1 3510779 yes (old gen) 3 no 0 0 0 0 0 > 2 3511915 yes (old gen) 2 no 0 0 0 0 0 > 3 402273 no 11 yes 12 13 0 0 0 > 4 81586 yes (old gen) 3 no 0 0 0 0 0 > 5 402274 no 9 yes 8 17 0 0 0 > 6 3614446 yes (old gen) 3 no 0 0 0 0 0 > 7 367731 yes (old gen) 4 no 0 0 0 0 0 > 8 367732 yes (old gen) 7 no 0 0 0 0 0 > 9 203769 yes (old gen) 4 no 0 0 0 0 0 > 10 178880 yes (old gen) 9 no 0 0 0 0 0 > 11 178881 yes (old gen) 13 no 0 0 0 0 0 > 12 3992043 yes (old gen) 8 no 0 0 0 0 0 > 13 3992044 yes (old gen) 3 no 0 0 0 0 0 > 14 4187117 yes (old gen) 5 no 0 0 0 0 0 > 15 4187118 yes (old gen) 6 no 0 0 0 0 0 > Sum 16 14 94 20 30 0 0 0 > > ..............G..................G..G............G..G...G....... > ..........._______WW_W__W_WWW_WW_WWW.........G...............___ > W_WWWW__WW_________W__GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG > GGGGG > ------------------------------------------------------------------------------- > mod_fcgid status: > > Total FastCGI processes: 0 > ------------------------------------------------------------------------------- > SSL/TLS Session Cache Status: > cache type: SHMCB, shared memory: 512000 bytes, current entries: 238 > subcaches: 32, indexes per subcache: 88 > time left on oldest entries' objects: avg: 57 seconds, (range: 1...195) > index usage: 8%, cache usage: 11% > total entries stored since starting: 51424 > total entries replaced since starting: 0 > total entries expired since starting: 51088 > total (pre-expiry) entries scrolled out of the cache: 0 > total retrieves since starting: 0 hit, 53 miss > total removes since starting: 98 hit, 4 miss > > > This is the current mpm settings > > <IfModule mpm_event_module> > StartServers 2 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadLimit 64 > ThreadsPerChild 25 > MaxRequestWorkers 150 > MaxConnectionsPerChild 0 > </IfModule> > > The cpu and server load seems to be fine. I read I should increment > MaxRequestWorkers, but where can I find how much I can increment? > The log said to increment ServerLimit but I didn't find this setting, > so I guess it got the default value.
Due to the graceful restart, these processes are trying to gracefully exit but being held alive presumably by websockets connections. You might look at tuning there, I am not sure what exists when mod_proxy is doing the tunnel or what the expected behavior is. To simply get more capacity, you would boost ServerLimit. The default is 16. Normally for event we want a little more than MaxRequestWorkers/ThreadsPerChild which 16 is already a good deal bigger than. I'd suggest doubling it for now and looking at server-status periodically. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
