I guess what you are saying is that the following error happens during startup and not during normal operation
( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting) Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Fri, Oct 22, 2021 at 2:23 PM Patrick Verdon <patrick.ver...@youreko.com> wrote: > Hi Yann, > > Quick update - we've enabled the core dumps but haven't been able to > reproduce the issue. After removing mod_http2 the first time we were able > to trigger the crash after 14 attempts but we've since tried over 100 times > with no luck. We'll keep trying as there's nothing worse than knowing > there's a bug lurking that can cause a crash. > > @Deepak - thanks for your suggestion but hitting MaxRequestWorkers is a > quirk of our installation, we load the max workers on startup so that the > PHP application is primed and ready, rather than have Apache spawn lots of > heavy processes. This is the same configuration we've had for years without > ever experiencing Apache hanging until the upgrade to 2.4.48. > > Thanks. > > Patrick > > *--* > > *Patrick Verdon | Founder* > Web: www.youreko.com > Mobile: +44 (0)7809 296438 > Skype: patrick_verdon > > This entire communication is sent on behalf of > Youreko Ltd and is strictly confidential to and > for the sole use of the intended addressee. > > Registered in England - 7448349 > > > > On Tue, 19 Oct 2021 at 11:00, Deepak Goel <deic...@gmail.com> wrote: > >> Hi >> >> Looks like the step 2 in your process is not working in the upgraded >> version of apache. >> >> Therefore it is vomiting out the following: >> server reached MaxRequestWorkers setting, consider raising the >> MaxRequestWorkers setting >> >> Deepak >> "The greatness of a nation can be judged by the way its animals are >> treated - Mahatma Gandhi" >> >> +91 73500 12833 >> deic...@gmail.com >> >> Facebook: https://www.facebook.com/deicool >> LinkedIn: www.linkedin.com/in/deicool >> >> "Plant a Tree, Go Green" >> >> Make In India : http://www.makeinindia.com/home >> >> >> On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon < >> patrick.ver...@youreko.com> wrote: >> >>> Hi All, >>> >>> I'd appreciate some feedback on an issue I'm experiencing. I've spent >>> quite some time researching the problem as it causes a serious outage in >>> our application. I've searched the Web, Stack Overflow, this list's mail >>> archives, the latest Apache bugs, and more, but have not been able to find >>> any reports of a similar issue. >>> >>> Background. I'm running the latest Apache 2.4.51 on Amazon Linux with >>> mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our >>> application take about 45 seconds to complete so there is a warm-up cache >>> procedure at regular intervals during the day which primes the varnish >>> cache. The following steps reliably cause Apache to hang, requiring a >>> manual restart: >>> >>> 1. Varnish cache is cleared, causing spike in load on httpd >>> 2. Warm-up cache process kicks off with 2 long running requests (45 >>> seconds each). This is a PHP application running under mod_php - each >>> process grows up to 700 MB, so the application kills the httpd child >>> process at the end to release the memory, using posix_kill(PID, 28). >>> 3. Apache hangs and does not recover. Varnish serves 503s. >>> 4. Manual restart required: service httpd restart >>> 5. Errors in the log show that 2 children had segmentation faults, >>> presumably the 2 with long running processes. >>> >>> >>> Albeit ugly, this process has been running for a year and a half without >>> any issues. We traced the date that crashes started to the date Apache was >>> upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an >>> issue in 2.4.51. >>> >>> See the error_log below and details about the installation. >>> >>> Any feedback on where to report this issue would be much appreciated. >>> >>> Thanks. >>> >>> Patrick >>> >>> -- >>> >>> # cat /var/log/httpd/error_log >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: >>> 0x0000557f94567e4f *** >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3166 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3483 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2657 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2660 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2661 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990579 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3172 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990592 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2681 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990603 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3254 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990615 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2685 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990627 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2688 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990639 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3015 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990652 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2696 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990664 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2699 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990680 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2710 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990692 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2713 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990703 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3250 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990716 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2721 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990726 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2724 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990739 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2734 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990750 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3471 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990769 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3109 exit signal Aborted (6) >>> [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2741 exit signal Segmentation fault (11) >>> *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: >>> 0x0000557f94567e4f *** >>> [Sun Oct 17 15:53:48.056539 2021] [core:notice] [pid 2620] AH00052: >>> child pid 3019 exit signal Aborted (6) >>> [Sun Oct 17 15:53:48.056584 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2707 exit signal Segmentation fault (11) >>> [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: >>> child pid 2727 exit signal Aborted (6) >>> [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] >>> AH00169: caught SIGTERM, shutting down >>> [Sun Oct 17 15:53:48.151770 2021] [suexec:notice] [pid 3575] AH01232: >>> suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) >>> [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The >>> mpm module (prefork.c) is not supported by mod_http2. The mpm determines >>> how things are processed in your server. HTTP/2 has more demands in this >>> regard and the currently selected mpm will just not do. This is an advisory >>> warning. Your server will continue to work, but the HTTP/2 protocol will be >>> inactive. >>> [Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] >>> AH02282: No slotmem from mod_heartmonitor >>> [Sun Oct 17 15:53:48.243891 2021] [mpm_prefork:notice] [pid 3581] >>> AH00163: Apache/2.4.51 (Amazon) OpenSSL/1.0.2k-fips configured -- resuming >>> normal operations >>> [Sun Oct 17 15:53:48.243923 2021] [core:notice] [pid 3581] AH00094: >>> Command line: '/usr/sbin/httpd' >>> [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] >>> AH00161: server reached MaxRequestWorkers setting, consider raising the >>> MaxRequestWorkers setting >>> >>> # httpd -V >>> Server version: Apache/2.4.51 (Amazon) >>> Server built: Oct 8 2021 19:30:47 >>> Server's Module Magic Number: 20120211:118 >>> Server loaded: APR 1.6.3, APR-UTIL 1.5.4 >>> Compiled using: APR 1.6.3, APR-UTIL 1.5.4 >>> Architecture: 64-bit >>> Server MPM: prefork >>> threaded: no >>> forked: yes (variable process count) >>> Server compiled with.... >>> -D APR_HAS_SENDFILE >>> -D APR_HAS_MMAP >>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>> -D APR_USE_SYSVSEM_SERIALIZE >>> -D APR_USE_PTHREAD_SERIALIZE >>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>> -D APR_HAS_OTHER_CHILD >>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>> -D DYNAMIC_MODULE_LIMIT=256 >>> -D HTTPD_ROOT="/etc/httpd" >>> -D SUEXEC_BIN="/usr/sbin/suexec" >>> -D DEFAULT_PIDLOG="/var/run/httpd/httpd.pid" >>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>> -D DEFAULT_ERRORLOG="logs/error_log" >>> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>> >>> # httpd -M >>> Loaded Modules: >>> core_module (static) >>> so_module (static) >>> http_module (static) >>> access_compat_module (shared) >>> actions_module (shared) >>> alias_module (shared) >>> allowmethods_module (shared) >>> auth_basic_module (shared) >>> auth_digest_module (shared) >>> authn_anon_module (shared) >>> authn_core_module (shared) >>> authn_dbd_module (shared) >>> authn_dbm_module (shared) >>> authn_file_module (shared) >>> authn_socache_module (shared) >>> authz_core_module (shared) >>> authz_dbd_module (shared) >>> authz_dbm_module (shared) >>> authz_groupfile_module (shared) >>> authz_host_module (shared) >>> authz_owner_module (shared) >>> authz_user_module (shared) >>> autoindex_module (shared) >>> cache_module (shared) >>> cache_disk_module (shared) >>> cache_socache_module (shared) >>> data_module (shared) >>> dbd_module (shared) >>> deflate_module (shared) >>> dir_module (shared) >>> dumpio_module (shared) >>> echo_module (shared) >>> env_module (shared) >>> expires_module (shared) >>> ext_filter_module (shared) >>> filter_module (shared) >>> headers_module (shared) >>> http2_module (shared) >>> include_module (shared) >>> info_module (shared) >>> log_config_module (shared) >>> logio_module (shared) >>> macro_module (shared) >>> mime_magic_module (shared) >>> mime_module (shared) >>> negotiation_module (shared) >>> remoteip_module (shared) >>> reqtimeout_module (shared) >>> request_module (shared) >>> rewrite_module (shared) >>> setenvif_module (shared) >>> slotmem_plain_module (shared) >>> slotmem_shm_module (shared) >>> socache_dbm_module (shared) >>> socache_memcache_module (shared) >>> socache_shmcb_module (shared) >>> status_module (shared) >>> substitute_module (shared) >>> suexec_module (shared) >>> unixd_module (shared) >>> userdir_module (shared) >>> version_module (shared) >>> vhost_alias_module (shared) >>> watchdog_module (shared) >>> dav_module (shared) >>> dav_fs_module (shared) >>> dav_lock_module (shared) >>> lua_module (shared) >>> mpm_prefork_module (shared) >>> proxy_module (shared) >>> lbmethod_bybusyness_module (shared) >>> lbmethod_byrequests_module (shared) >>> lbmethod_bytraffic_module (shared) >>> lbmethod_heartbeat_module (shared) >>> proxy_ajp_module (shared) >>> proxy_balancer_module (shared) >>> proxy_connect_module (shared) >>> proxy_express_module (shared) >>> proxy_fcgi_module (shared) >>> proxy_fdpass_module (shared) >>> proxy_ftp_module (shared) >>> proxy_http_module (shared) >>> proxy_hcheck_module (shared) >>> proxy_scgi_module (shared) >>> proxy_uwsgi_module (shared) >>> proxy_wstunnel_module (shared) >>> ssl_module (shared) >>> cgi_module (shared) >>> php7_module (shared) >>> >>> # yum list | grep mod_ >>> lighttpd-mod_authn_gssapi.x86_64 1.4.53-1.36.amzn1 >>> amzn-updates >>> lighttpd-mod_authn_mysql.x86_64 1.4.53-1.36.amzn1 >>> amzn-updates >>> lighttpd-mod_authn_pam.x86_64 1.4.53-1.36.amzn1 >>> amzn-updates >>> lighttpd-mod_geoip.x86_64 1.4.53-1.36.amzn1 >>> amzn-updates >>> lighttpd-mod_mysql_vhost.x86_64 1.4.53-1.36.amzn1 >>> amzn-updates >>> mod_auth_kerb.x86_64 5.4-10.9.amzn1 >>> amzn-main >>> mod_auth_mellon.x86_64 0.13.1-1.6.amzn1 >>> amzn-updates >>> mod_auth_mysql.x86_64 1:3.0.0-18.10.amzn1 >>> amzn-main >>> mod_auth_pgsql.x86_64 2.0.3-10.1.5.amzn1 >>> amzn-main >>> mod_authz_ldap.x86_64 0.26-16.8.amzn1 >>> amzn-main >>> mod_dav_svn.x86_64 1.9.7-1.54.amzn1 >>> amzn-main >>> mod_fcgid.x86_64 2.3.9-1.6.amzn1 >>> amzn-main >>> mod_geoip.x86_64 1.2.7-1.2.amzn1 >>> amzn-main >>> mod_nss.x86_64 1.0.10-1.13.amzn1 >>> amzn-main >>> mod_perl.x86_64 2.0.7-7.28.amzn1 >>> amzn-updates >>> mod_perl-devel.x86_64 2.0.7-7.28.amzn1 >>> amzn-updates >>> mod_proxy_html.x86_64 3.1.2-7.3.amzn1 >>> amzn-main >>> mod_python26.x86_64 3.3.1-17.20.amzn1 >>> amzn-main >>> mod_python27.x86_64 3.3.1-17.20.amzn1 >>> amzn-main >>> mod_security.x86_64 2.8.0-5.27.amzn1 >>> amzn-main >>> mod_security_crs.noarch 2.2.8-2.5.amzn1 >>> amzn-main >>> mod_security_crs-extras.noarch 2.2.8-2.5.amzn1 >>> amzn-main >>> mod_ssl.x86_64 1:2.2.34-1.16.amzn1 >>> amzn-main >>> mod_wsgi-python26.x86_64 3.2-6.12.amzn1 >>> amzn-updates >>> mod_wsgi-python27.x86_64 3.2-6.12.amzn1 >>> amzn-updates >>> >>> *--* >>> >>> *Patrick Verdon | Founder* >>> Web: www.youreko.com >>> Mobile: +44 (0)7809 296438 >>> Skype: patrick_verdon >>> >>> This entire communication is sent on behalf of >>> Youreko Ltd and is strictly confidential to and >>> for the sole use of the intended addressee. >>> >>> Registered in England - 7448349 >>> >>> >>
