Hi,
We've been having some issues with our mod_perl web servers recently. Basically what happens is the server becomes very very slow, to the point of being almost totally unresponsive. At the time this happens, we see this in the error log:
[Thu Sep 30 09:44:43 2004] [error] (12)Cannot allocate memory: fork: Unable to fork new process
This leaves us with one Apache child running, which doesn't respond to requests. The only way to restore service to the machine is to kill Apache manually then start it up again.
This may be related, but at the time this occurs mod_perl seems to have difficulty getting the date from the server, i.e. we get this in our error logs as well:
[Thu Sep 30 00:20:20 2004] [error] Hour '2004-- 00' out of range 0..23 at /www/cgi-bin/Shared/DateStuff.pm line 1111\n
This is because a shell call to date ($date = `date`;) is failing - this is most likely what's causing the failure to fork we see above.
We're running Fedora Core 1 on IBM eServers with a custom-built mod_perl httpd. Apache is built with the following command:
perl Makefile.PL EVERYTHING=1 USE_APACI=1 DYNAMIC=1 APACHE_PREFIX=/www/ APACHE_SRC=/usr/local/src/apache_1.3.31/src DO_HTTPD=1 APACI_ARGS="--enable-shared=max"
make && make test && make install
This is what we saw last time the problem happened (this morning):
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd nobody 25661 5.0 12.4 749172 128132 ? S 09:54 0:43 /www/bin/httpd nobody 26374 3.9 8.2 705296 85016 ? S 09:57 0:28 /www/bin/httpd nobody 28181 4.2 8.2 707132 85600 ? S 10:03 0:13 /www/bin/httpd nobody 28480 4.7 8.2 706288 84652 ? S 10:04 0:12 /www/bin/httpd nobody 28713 3.8 7.0 694656 72312 ? S 10:05 0:08 /www/bin/httpd nobody 28920 3.8 7.6 700608 78600 ? S 10:05 0:07 /www/bin/httpd nobody 29013 3.8 7.0 694076 72320 ? S 10:06 0:06 /www/bin/httpd nobody 29109 3.8 7.1 696480 73940 ? S 10:06 0:05 /www/bin/httpd nobody 29158 4.1 7.1 695772 73780 ? S 10:06 0:05 /www/bin/httpd nobody 29557 4.1 6.4 689356 66168 ? S 10:07 0:02 /www/bin/httpd nobody 29596 5.0 6.2 687308 64420 ? S 10:08 0:02 /www/bin/httpd root 29880 0.0 0.0 5580 572 pts/0 R 10:08 0:00 grep ht
[EMAIL PROTECTED] root]# /www/bin/apachectl stop /www/bin/apachectl stop: httpd stopped
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd nobody 25661 5.0 0.0 0 0 ? Z 09:54 0:43 [httpd <defunct>] nobody 26374 4.0 0.0 0 0 ? Z 09:57 0:28 [httpd <defunct>] nobody 28181 4.3 0.0 0 0 ? Z 10:03 0:13 [httpd <defunct>] nobody 28480 4.7 8.8 706288 91780 ? R 10:04 0:12 /www/bin/httpd nobody 28713 3.9 7.0 695188 72812 ? R 10:05 0:08 /www/bin/httpd nobody 29013 3.9 0.0 0 0 ? Z 10:06 0:06 [httpd <defunct>] nobody 29109 3.9 0.0 0 0 ? Z 10:06 0:05 [httpd <defunct>] nobody 29158 4.4 0.0 0 0 ? RW 10:06 0:06 [httpd] nobody 29557 4.2 0.0 0 0 ? Z 10:07 0:02 [httpd <defunct>] nobody 29596 5.2 0.0 0 0 ? Z 10:08 0:03 [httpd <defunct>] nobody 29909 0.0 7.0 695188 72812 ? R 10:09 0:00 /www/bin/httpd root 29911 0.0 0.0 4852 576 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd nobody 28713 3.9 7.4 695188 77176 ? S 10:05 0:08 /www/bin/httpd nobody 29909 1.0 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd root 29913 0.0 0.0 4516 576 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd nobody 28713 3.8 7.4 695188 77176 ? S 10:05 0:08 /www/bin/httpd nobody 29909 0.6 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd root 29915 0.0 0.0 4720 576 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 7619 0.0 4.5 677332 46944 ? S Sep24 2:06 /www/bin/httpd nobody 28713 3.8 7.4 695188 77176 ? S 10:05 0:08 /www/bin/httpd nobody 29909 0.5 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd root 29917 0.0 0.0 4856 560 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# killall httpd
[EMAIL PROTECTED] root]# ps -auxww | grep ht nobody 29909 0.2 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd root 29923 0.0 0.0 4528 576 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# ps -auxww | grep ht nobody 29909 0.2 7.4 695188 77176 ? S 10:09 0:00 /www/bin/httpd root 29925 0.0 0.0 5384 572 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# kill -9 29909
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 29930 0.0 0.0 4776 576 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# /www/bin/apachectl start /www/bin/apachectl start: httpd started
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 29946 92.0 6.0 68676 62348 ? R 10:09 0:00 /www/bin/httpd root 29968 0.0 0.0 4964 576 pts/0 R 10:09 0:00 grep ht
[EMAIL PROTECTED] root]# ps -auxww | grep ht root 29946 68.0 7.0 78476 72312 ? S 10:09 0:01 /www/bin/httpd nobody 29971 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29972 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29973 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29974 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29975 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29976 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29977 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29978 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29979 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd nobody 29980 0.0 7.0 78476 72324 ? S 10:09 0:00 /www/bin/httpd root 29982 0.0 0.0 5052 572 pts/0 R 10:09 0:00 grep ht
Any ideas as to what might be causing this problem would be greatly appreciated.
Regards,
-- Alex
-- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
