Hello,
In case something hogs the workers, until you figure out what and solve it:
MaxConnectionsPerChild 0 # inifinite connections per child so apache
does not try to gracefully shut them down.
MaxSpareThreads = MaxRequestWorkers
Never do a graceful restart.
This is what I had to do in production with certain older version of
apache in which mod_weblogic would cause the same behaviour you
describe
Cheers
El lun, 28 nov 2022 a las 5:28, Mike Dewhirst
(<mi...@dewhirst.com.au>) escribió:
>
> On 28/11/2022 12:09 pm, Mike Dewhirst wrote:
>
> On 26/11/2022 11:41 am, Mike Dewhirst wrote:
>
> On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
>
> Some script is probably hogging the workers and not letting them close so if
> you must, just restart, but judging by the amount of traffic you seem to have
> at the time I would truncate in logrotate and not restart or reload the
> server at all.
>
>
> I run a Python script on the production machine to delete the entire site
> (including directory structure) then redeploy everything from the repo
> whenever anything changes.
>
> On the basis that Apache (mod-wsgi) only reads the files once on reload, if
> they also had the dead finger of my script or perhaps the Python interpreter
> hanging on, that may well have "hogged" the workers.
>
> That script now restarts Apache instead of reload.
>
> I have now adjusted the logrotate script to avoid reload/restart logging
> status as follows:
>
> /var/log/apache2/*.log {
> daily
> missingok
> rotate 14
> compress
> delaycompress
> notifempty
> copytruncate
> sharedscripts
> prerotate
> if [ -d /etc/logrotate.d/httpd-prerotate ]; then
> run-parts /etc/logrotate.d/httpd-prerotate
> fi
> endscript
> postrotate
> if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
> invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
> fi
> endscript
> }
>
>
> ... and left the mpm-event settings as follows:
>
> <IfModule mpm_event_module>
> ServerLimit 32 # Thanks Eric
> Covener
> StartServers 16
> MinSpareThreads 400
> MaxSpareThreads 250
> ThreadLimit 64
> ThreadsPerChild 50
> AsyncRequestWorkerFactor 2
> MaxRequestWorkers 800
> MaxConnectionsPerChild 0
> </IfModule>
>
> I restarted the entire machine
>
> Next logrotate is 00:00 UTC -- I'll report tomorrow.
>
>
> Zero bytes in the error log since Saturday when I sent the above email.
>
> That covers the period with copytruncate in the Apache logrotate conf.
>
> I haven't updated the site so no restarts from me.
>
> I'll adjust the deployment script back to reload instead of restart before my
> next deployment and report back.
>
>
> Errors recommence as follows:
>
> ... not an error but the reload after a deployment script run...
>
> [Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0
> Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid 140638735415168]
> AH00094: Command line: '/usr/sbin/apache2'
>
> ... lengthy period ensues with no errors followed by another deployment
> reload ...
>
> [Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0
> Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid 140638735415168]
> AH00094: Command line: '/usr/sbin/apache2'
>
> ... another deployment script reload after a period with no errors ...
>
> [Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0
> Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid 140638735415168]
> AH00094: Command line: '/usr/sbin/apache2'
>
> ... but two seconds later ...
>
> [Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid
> 140638735415168] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
>
> ... followed by more of the same at approximately 1 second intervals ...
>
> [Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid
> 140638735415168] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
>
> About to revert my deployment script to restart Apache instead of reload ...
>
> Not sure what else I can do?
>
> Cheers
>
> Mike
>
>
>
> Cheers
>
> Mike
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org
