Bug#779077: apache2-bin: crash with segmentation fault if gracefully reloaded twice too quickly

[Chris Boot]

Package: apache2-bin
Version: 2.4.10-9
Severity: important

Dear maintainers,

When reloading apache2 gracefully (/etc/init.d/apache2 reload) twice in
quick succession, the parent process can crash. Below are the
backtraces from 3 core dumps several days in a row that happened during
logrotate.

We had set up two logrotate jobs for Apache, one was the default one as
shipped with the package, the other was to rotate some additional
per-virtual-host logs that are not kept under /var/log. With both
rotation jobs reloading Apache, the process would crash and leave the
web server inaccessible.

It appears as though this may be caused by the following chain of
events:
1. apache2 is reloaded and signals children to stop to be cycled
2. a request comes in for a PHP script, which is run via mod_fcgid on
   this server; a subprocess spawns
3. before all the children for the previous generation have been
   stopped, the server is reloaded again

I have unfortunately not managed to replicate this on any other Apache
server that we maintain. This server is running Jessie and is using the
worker MPM (not event).

Regards,
Chris

-- Package-specific info:


*** apache2.core.20150221-0045.txt
[New LWP 1218]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7fbd2b1f8e10 in ?? ()
#0  0x7fbd2b1f8e10 in ?? ()
No symbol table info available.
#1  
No locals.
#2  0x7fbd2e6b147c in __libc_waitpid (pid=1222,
stat_loc=stat_loc@entry=0x7fff79f676ac, options=options@entry=2) at
../sysdeps/unix/sysv/linux/waitpid.c:31
resultvar = 18446744073709551612
oldtype = 2046195392
#3  0x7fbd2e8e620b in apr_proc_wait (proc=0x7fbd2f046318,
exitcode=0x7fff79f676b0, exitcode@entry=0x0, exitwhy=0x7fff79f676b4,
exitwhy@entry=0x0, waithow=waithow@entry=APR_WAIT) at
/tmp/buildd/apr-1.5.1/threadproc/unix/proc.c:633
pstatus = 
waitpid_options = 2
exit_int = 15
ignore = 15
ignorewhy = APR_PROC_SIGNAL
#4  0x7fbd2e8dac6e in free_proc_chain (procs=) at
/tmp/buildd/apr-1.5.1/memory/unix/apr_pools.c:2519
pc = 0x7fbd2f046338
need_timeout = 
timeout_interval = 
#5  0x7fbd2e8dbca0 in apr_pool_clear (pool=0x7fbd2f19a028) at
/tmp/buildd/apr-1.5.1/memory/unix/apr_pools.c:777
active = 
#6  0x7fbd2f1e5068 in main (argc=3, argv=0x7fff79f67908) at main.c:707
c = 0 '\000'
error = 0xfffc 
process = 0x7fbd2f1a3118
pconf = 0x7fbd2f19a028
plog = 0x7fbd2f16e028
ptemp = 0x7fbd2f170028
pcommands = 0x7fbd2f178028
opt = 0x7fbd2f178118
mod = 0x7fbd2f44a1c0 
opt_arg = 0x7fbd2f1a3028 "(p\032/\275\177"
signal_server = 0xfffc

*** apache2.core.20150222-0035.txt
[New LWP 12542]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f2513947e10 in ?? ()
#0  0x7f2513947e10 in ?? ()
No symbol table info available.
#1  
No locals.
#2  0x7f2516e0047c in __libc_waitpid (pid=12548,
stat_loc=stat_loc@entry=0x7fff56c434dc, options=options@entry=2) at
../sysdeps/unix/sysv/linux/waitpid.c:31
resultvar = 18446744073709551612
oldtype = 1455699184
#3  0x7f251703520b in apr_proc_wait (proc=0x7f2517795318,
exitcode=0x7fff56c434e0, exitcode@entry=0x0, exitwhy=0x7fff56c434e4,
exitwhy@entry=0x0, waithow=waithow@entry=APR_WAIT) at
/tmp/buildd/apr-1.5.1/threadproc/unix/proc.c:633
pstatus = 
waitpid_options = 2
exit_int = 15
ignore = 15
ignorewhy = APR_PROC_SIGNAL
#4  0x7f2517029c6e in free_proc_chain (procs=) at
/tmp/buildd/apr-1.5.1/memory/unix/apr_pools.c:2519
pc = 0x7f2517795338
need_timeout = 
timeout_interval = 
#5  0x7f251702aca0 in apr_pool_clear (pool=0x7f25178e9028) at
/tmp/buildd/apr-1.5.1/memory/unix/apr_pools.c:777
active = 
#6  0x7f2517934068 in main (argc=3, argv=0x7fff56c43738) at main.c:707
c = 0 '\000'
error = 0xfffc 
process = 0x7f25178f2118
pconf = 0x7f25178e9028
plog = 0x7f25178bd028
ptemp = 0x7f25178bf028
pcommands = 0x7f25178c7028
opt = 0x7f25178c7118
mod = 0x7f2517b991c0 
opt_arg = 0x7f25178f2028 "(`\217\027%\177"
signal_server = 0xfffc

*** apache2.core.20150223-0040.txt
[New LWP 23405]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f094169ee10 in ?? ()
#0  0x7f094169ee10

Bug#779078: apache2-bin: event mpm: child segfault in notify_suspend causes parent to exit during log rotation

[Chris Boot]

Package: apache2-bin
Version: 2.4.10-9
Severity: important

Dear maintainers,

We have been experiencing segmentation faults in apache2 when using the
event MPM in jessie. These manifest themselves with log entries that
look like:

[Thu Feb 19 14:26:22.477619 2015] [core:notice] [pid 14018:tid
140488395114368] AH00052: child pid 14024 exit signal Segmentation fault
(11)
[Thu Feb 19 19:26:22.282410 2015] [core:notice] [pid 19271:tid
140434662569856] AH00052: child pid 19276 exit signal Segmentation fault
(11)
[Thu Feb 19 19:35:33.842016 2015] [core:notice] [pid 19271:tid
140434662569856] AH00052: child pid 15360 exit signal Segmentation fault
(11)

When logs are rotated nightly and apache2 is instructed to reopen its
logs, the parent process exits altogether:

[Fri Feb 20 00:52:14.175939 2015] [mpm_event:notice] [pid 19271:tid
140434662569856] AH00493: SIGUSR1 received.  Doing graceful restart
[Fri Feb 20 00:52:15.279574 2015] [core:notice] [pid 19271] AH00060: seg
fault or similar nasty error detected in the parent process

We have enabled core dumps and managed to capture one crash (so far)
when a child process crashes, which is included below as
apache2.core.20150220-1158.txt (output from gdb with debug packages
installed).

I will attempt to obtain and analyse further core dumps, but our client
may prefer to switch to the worker MPM in order to keep their sites
working, in which case I will be limited in what more I can do.

I have set severity important because this is a crash in the default
MPM for jessie, and our client's environment doesn't strike me as
particularly unusual so this may well affect many users.

Regards,
Chris

-- Package-specific info:


*** loaded modules
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgid_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 fcgid_module (shared)
 filter_module (shared)
 mime_module (shared)
 mpm_event_module (shared)
 negotiation_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 socache_shmcb_module (shared)
 ssl_module (shared)
 status_module (shared)

*** apache2.core.20150220-1158.txt
12:58 997236 ~ # gdb /usr/sbin/apache2 apache2.core.20150220-1158
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...Reading symbols from
/usr/lib/debug//usr/sbin/apache2...done.
done.
[New LWP 385]
[New LWP 380]
[New LWP 395]
[New LWP 389]
[New LWP 386]
[New LWP 394]
[New LWP 397]
[New LWP 388]
[New LWP 390]
[New LWP 392]
[New LWP 383]
[New LWP 420]
[New LWP 399]
[New LWP 424]
[New LWP 413]
[New LWP 402]
[New LWP 422]
[New LWP 416]
[New LWP 414]
[New LWP 403]
[New LWP 408]
[New LWP 409]
[New LWP 406]
[New LWP 411]
[New LWP 405]
[New LWP 418]
[New LWP 384]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f3e1fa57ab2 in notify_suspend (cs=) at event.c:887
887 event.c: No such file or directory.
(gdb) thread apply all bt full

Thread 27 (Thread 0x7f3e1d422700 (LWP 384)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x7f3e2313977d in apr_thread_cond_wait (cond=,
mutex=) at /tmp/buildd/apr-1.5.1/locks/unix/thread_cond.c:68
No locals.
#2  0x7f3e1fa5b205 in ap_queue_pop_something (queue=0x7f3e239575c8,
sd=0x7f3e1d421e70, ecs=0x7f3e1d421e78, p=0x7f3e1d421e80,
te_out=0x7f3e1d421e88) at fdqueue.c:438
elem = 
rv = 
#3  0x7f3e1fa5778f in worker_thread (thd=0x7f3e23957664, dummy=0x80)
at event.c:1823
thread_slot = 1
csd = 0x7f3e0c61f0b0
cs = 0x7f3e0c61f2b8
ptrans = 0x7f3e0c61f028
is_idle = 0
te = 0x0
#4  0x7f3e22f090a4 in start_thread (arg=0x7f3e1d422700) at
pthread_creat

Bug#779077: apache2-bin: crash with segmentation fault if gracefully reloaded twice too quickly

[Chris Boot]

Hi,

We've just hit the same crash again, but on a different server for a
different client of ours. This time it was an upgrade from Wheezy to
Jessie, but the MPM is still worker.

This time in particular we hit the crash during the phpmyadmin security
upgrade recently. If I read the phpmyadmin maintscripts correctly,
Apache is reloaded once automatically by the apache2-maintscript-helper
when apache2_invoke is called to enable/disable its configuration file,
and once by the phpmyadmin maintscript itself via invoke-rc.d.

Best regards,
Chris

-- 
Chris Boot

Tiger Computing Ltd
IS27001:2013 Certified

Tel: 01600 483 484
Web: https://www.tiger-computing.co.uk

Registered in England. Company number: 3389961
Registered address: Wyastone Business Park,
 Wyastone Leys, Monmouth, NP25 3SR