On 3/29/2021 8:45 AM, A.J. Gatlin wrote:
Hi there,
I'm running some PHP-based sites on CentOS 8, which uses version
2.4.37 of Apache. Apache is set to use MPM Event mode, which is also
the default.
Once per day during a quiet period, logrotate rotates the Apache logs
by doing a graceful restart (reload) as part of a postrotate command:
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
This causes the following line to appear in the Apache error log,
which I believe to be normal and expected:
[mpm_event:notice] [pid 1413:tid 140292271810880] AH00493: SIGUSR1
received. Doing graceful restart
The log is then rotated, and in the newly created error log, I see the
following:
[lbmethod_heartbeat:notice] [pid 1413:tid 140292271810880] AH02282: No
slotmem from mod_heartmonitor
[mpm_event:notice] [pid 1413:tid 140292271810880] AH00489:
Apache/2.4.37 (centos) OpenSSL/1.1.1g configured -- resuming normal
operations
[core:notice] [pid 1413:tid 140292271810880] AH00094: Command line:
'/usr/sbin/httpd -D FOREGROUND'
Then, immediately after, I see this error repeated once per second:
[mpm_event:error] [pid 1413:tid 140292271810880] AH03490: scoreboard
is full, not at MaxRequestWorkers.Increase ServerLimit.
The server then becomes unresponsive and will not answer HTTPS
requests. It continues logging the above error and will not begin
working again until I manually restart the Apache process.
All configs on this server are essentially the default for CentOS 8. I
have not tried making any adjustments to the MPM Event config settings
-- mostly because I'm not sure what setting for ServerLimit would be
"correct," and also because this problem happens *only* when logrotate
does its graceful restart, and not during other periods of high traffic.
My research on this issue brings up a lot of results about an old
Apache bug related to MPM Event mode, but my version of Apache
shouldn't be affected by this particular bug. I have also seen
suggestions that some long-running processes may hang and eat up all
available worker processes, but I don't understand why this would
interrupt the restart. And I don't have any particularly long-running
processes served by Apache -- just quickly executing PHP scripts.
Can anyone suggest what might fix this issue, given that it seems to
happen only during a graceful reload? Should I try changing the method
that logrotate uses to restart Apache?
Is there some other way I can avoid the server hanging, or is it
really necessary to increase the ServerLimit as the error suggests?
Thank you very much for any insight or help.
Have you checked your request logs at the time of restart for any
request activity associated with the restart?
Does https://yourserver.com/server-status present any clues,
particularly the processes section or will that also not respond?
Does a restart, graceful or full restart, outside of a logrotate present
the same issue?
Maybe give MaxConnectionsPerChild a setting. I believe the default is 0
which indicates httpd children do not expire. Maybe try 2000 so httpd
children expire occasionally and restart new... just in case of some
memory issue you are facing.
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxconnectionsperchild
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org
