The gdb from some of the processes are: -------- [root@prd-web2887652808b bin]# gdb httpd 23668 GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-30.amzn2.0.3 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done. done. Attaching to program: /usr/sbin/httpd, process 23668 Reading symbols from target:/lib64/libpcre.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libpcre.so.1.2.0.debug...done. done. Reading symbols from target:/lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libselinux.so.1.debug...done. done. Reading symbols from target:/lib64/libaprutil-1.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libaprutil-1.so.0.6.1.debug...done. done. Reading symbols from target:/lib64/libcrypt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libcrypt-2.26.so.debug...done. done. Reading symbols from target:/lib64/libexpat.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libexpat.so.1.6.0.debug...done. done. Reading symbols from target:/lib64/libapr-1.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libapr-1.so.0.6.3.debug...done. done. Reading symbols from target:/lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libpthread-2.26.so.debug...done. done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Reading symbols from target:/lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.26.so.debug...done. done. Reading symbols from target:/lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.26.so.debug...done. done. Reading symbols from target:/lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.26.so.debug...done. done. Reading symbols from target:/lib64/libuuid.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libuuid.so.1.3.0.debug...done. done. Reading symbols from target:/lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.26.so.debug...done. done. Reading symbols from target:/lib64/libnss_sss.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libnss_sss.so.2.debug...done. done. Reading symbols from target:/lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libnss_files-2.26.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_access_compat.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_access_compat.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_actions.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_actions.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_alias.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_allowmethods.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_allowmethods.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_authn_core.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_core.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_authz_core.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_core.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_authz_host.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_host.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_deflate.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_deflate.so.debug...done. done. Reading symbols from target:/lib64/libz.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_dir.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dir.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_dumpio.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dumpio.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_env.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_env.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_expires.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_expires.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_filter.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_filter.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_headers.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_headers.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_include.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_include.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_log_config.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_log_config.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_logio.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_logio.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_mime_magic.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mime_magic.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_mime.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mime.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_negotiation.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_negotiation.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_remoteip.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_remoteip.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_reqtimeout.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_reqtimeout.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_request.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_request.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_rewrite.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_rewrite.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_setenvif.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_setenvif.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_slotmem_shm.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_slotmem_shm.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_socache_shmcb.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_socache_shmcb.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_substitute.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_substitute.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_unixd.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_unixd.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_vhost_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_vhost_alias.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_watchdog.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_watchdog.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_mpm_worker.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mpm_worker.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_connect.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_connect.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_express.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_express.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_http.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_systemd.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_systemd.so.debug...done. done. Reading symbols from target:/lib64/libsystemd.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsystemd.so.0.6.0.debug...done. done. Reading symbols from target:/lib64/libcap.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libcap.so.2.22.debug...done. done. Reading symbols from target:/lib64/libm.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libm-2.26.so.debug...done. done. Reading symbols from target:/lib64/liblzma.so.5...Reading symbols from /usr/lib/debug/usr/lib64/liblzma.so.5.2.2.debug...done. done. Reading symbols from target:/lib64/liblz4.so.1...Reading symbols from /usr/lib/debug/usr/lib64/liblz4.so.1.7.5.debug...done. done. Reading symbols from target:/lib64/libgcrypt.so.11...Reading symbols from /usr/lib/debug/usr/lib64/libgcrypt.so.11.8.2.debug...done. done. Reading symbols from target:/lib64/libgpg-error.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libgpg-error.so.0.10.0.debug...done. done. Reading symbols from target:/lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libresolv-2.26.so.debug...done. done. Reading symbols from target:/lib64/libdw.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libdw-0.176.so.debug...done. done. Reading symbols from target:/lib64/libgcc_s.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libgcc_s-7-20180712.so.1.debug...done. done. Reading symbols from target:/lib64/libattr.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libattr.so.1.1.0.debug...done. done. Reading symbols from target:/lib64/libelf.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libelf-0.176.so.debug...done. done. Reading symbols from target:/lib64/libbz2.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libbz2.so.1.0.6.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_http2.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_http2.so.debug...done. done. Reading symbols from target:/lib64/libnghttp2.so.14...Reading symbols from /usr/lib/debug/usr/lib64/libnghttp2.so.14.20.0.debug...done. done. Reading symbols from target:/lib64/libcrypto.so.10...Reading symbols from /usr/lib/debug/usr/lib64/libcrypto.so.1.0.2k.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_http2.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http2.so.debug...done. done. get_next (slot=0x5591071acd18) at h2_workers.c:195 195 status = h2_fifo_try_peek(workers->mplxs, mplx_peek, slot); (gdb) where #0 get_next (slot=0x5591071acd18) at h2_workers.c:195 #1 slot_run (thread=0x5591072441e0, wctx=0x5591071acd18) at h2_workers.c:228 #2 0x00007fcf0690640b in start_thread (arg=0x7fced6bf5700) at pthread_create.c:465 #3 0x00007fcf0643d09f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 -------- [root@prd-web2887652808b bin]# gdb httpd 23660 GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-30.amzn2.0.3 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done. done. Attaching to program: /usr/sbin/httpd, process 23660 Reading symbols from target:/lib64/libpcre.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libpcre.so.1.2.0.debug...done. done. Reading symbols from target:/lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libselinux.so.1.debug...done. done. Reading symbols from target:/lib64/libaprutil-1.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libaprutil-1.so.0.6.1.debug...done. done. Reading symbols from target:/lib64/libcrypt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libcrypt-2.26.so.debug...done. done. Reading symbols from target:/lib64/libexpat.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libexpat.so.1.6.0.debug...done. done. Reading symbols from target:/lib64/libapr-1.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libapr-1.so.0.6.3.debug...done. done. Reading symbols from target:/lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libpthread-2.26.so.debug...done. done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Reading symbols from target:/lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.26.so.debug...done. done. Reading symbols from target:/lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.26.so.debug...done. done. Reading symbols from target:/lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.26.so.debug...done. done. Reading symbols from target:/lib64/libuuid.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libuuid.so.1.3.0.debug...done. done. Reading symbols from target:/lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.26.so.debug...done. done. Reading symbols from target:/lib64/libnss_sss.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libnss_sss.so.2.debug...done. done. Reading symbols from target:/lib64/libnss_files.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libnss_files-2.26.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_access_compat.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_access_compat.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_actions.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_actions.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_alias.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_allowmethods.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_allowmethods.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_authn_core.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authn_core.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_authz_core.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_core.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_authz_host.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_authz_host.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_deflate.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_deflate.so.debug...done. done. Reading symbols from target:/lib64/libz.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_dir.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dir.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_dumpio.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_dumpio.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_env.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_env.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_expires.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_expires.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_filter.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_filter.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_headers.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_headers.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_include.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_include.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_log_config.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_log_config.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_logio.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_logio.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_mime_magic.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mime_magic.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_mime.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mime.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_negotiation.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_negotiation.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_remoteip.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_remoteip.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_reqtimeout.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_reqtimeout.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_request.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_request.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_rewrite.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_rewrite.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_setenvif.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_setenvif.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_slotmem_shm.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_slotmem_shm.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_socache_shmcb.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_socache_shmcb.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_substitute.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_substitute.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_unixd.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_unixd.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_vhost_alias.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_vhost_alias.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_watchdog.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_watchdog.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_mpm_worker.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_mpm_worker.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_connect.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_connect.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_express.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_express.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_http.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http.so.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_systemd.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_systemd.so.debug...done. done. Reading symbols from target:/lib64/libsystemd.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsystemd.so.0.6.0.debug...done. done. Reading symbols from target:/lib64/libcap.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libcap.so.2.22.debug...done. done. Reading symbols from target:/lib64/libm.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libm-2.26.so.debug...done. done. Reading symbols from target:/lib64/liblzma.so.5...Reading symbols from /usr/lib/debug/usr/lib64/liblzma.so.5.2.2.debug...done. done. Reading symbols from target:/lib64/liblz4.so.1...Reading symbols from /usr/lib/debug/usr/lib64/liblz4.so.1.7.5.debug...done. done. Reading symbols from target:/lib64/libgcrypt.so.11...Reading symbols from /usr/lib/debug/usr/lib64/libgcrypt.so.11.8.2.debug...done. done. Reading symbols from target:/lib64/libgpg-error.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libgpg-error.so.0.10.0.debug...done. done. Reading symbols from target:/lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libresolv-2.26.so.debug...done. done. Reading symbols from target:/lib64/libdw.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libdw-0.176.so.debug...done. done. Reading symbols from target:/lib64/libgcc_s.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libgcc_s-7-20180712.so.1.debug...done. done. Reading symbols from target:/lib64/libattr.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libattr.so.1.1.0.debug...done. done. Reading symbols from target:/lib64/libelf.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libelf-0.176.so.debug...done. done. Reading symbols from target:/lib64/libbz2.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libbz2.so.1.0.6.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_http2.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_http2.so.debug...done. done. Reading symbols from target:/lib64/libnghttp2.so.14...Reading symbols from /usr/lib/debug/usr/lib64/libnghttp2.so.14.20.0.debug...done. done. Reading symbols from target:/lib64/libcrypto.so.10...Reading symbols from /usr/lib/debug/usr/lib64/libcrypto.so.1.0.2k.debug...done. done. Reading symbols from target:/etc/httpd/modules/mod_proxy_http2.so...Reading symbols from /usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http2.so.debug...done. done. get_next (slot=0x5591071acd98) at h2_workers.c:196 196 if (status == APR_EOF) { (gdb) where #0 get_next (slot=0x5591071acd98) at h2_workers.c:196 #1 slot_run (thread=0x559107244030, wctx=0x5591071acd98) at h2_workers.c:228 #2 0x00007fcf0690640b in start_thread (arg=0x7fced7bf7700) at pthread_create.c:465 #3 0x00007fcf0643d09f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 --------- fifo_peek (block=0, ctx=ctx@entry=0x5591071acf58, fn=fn@entry=0x7fceff358fd0 <mplx_peek>, fifo=0x5591071ac7a0) at h2_util.c:767 767 return APR_EOF; (gdb) where #0 fifo_peek (block=0, ctx=ctx@entry=0x5591071acf58, fn=fn@entry=0x7fceff358fd0 <mplx_peek>, fifo=0x5591071ac7a0) at h2_util.c:767 #1 h2_fifo_try_peek (fifo=0x5591071ac7a0, fn=fn@entry=0x7fceff358fd0 <mplx_peek>, ctx=ctx@entry=0x5591071acf58) at h2_util.c:792 #2 0x00007fceff358e3d in get_next (slot=0x5591071acf58) at h2_workers.c:195 #3 slot_run (thread=0x559107243a48, wctx=0x5591071acf58) at h2_workers.c:228 #4 0x00007fcf0690640b in start_thread (arg=0x7fcedb3fe700) at pthread_create.c:465 #5 0x00007fcf0643d09f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ----------- slot_run (thread=0x559107243da8, wctx=0x5591071ace58) at h2_workers.c:225 225 while (!slot->aborted) { (gdb) where #0 slot_run (thread=0x559107243da8, wctx=0x5591071ace58) at h2_workers.c:225 #1 0x00007fcf0690640b in start_thread (arg=0x7fced93fa700) at pthread_create.c:465 #2 0x00007fcf0643d09f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ------------ I also ran gdb on the parent process: Attaching to program: /usr/sbin/httpd, process 22231 [New LWP 23620] [New LWP 23636] [New LWP 23648] [New LWP 23660] [New LWP 23664] [New LWP 23668] [New LWP 23677] [New LWP 23698] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007fcf0690bd63 in futex_wait (private=<optimized out>, expected=12, futex_word=0x5591072435e4) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 61 int err = lll_futex_timed_wait (futex_word, expected, NULL, private); (gdb) where #0 0x00007fcf0690bd63 in futex_wait (private=<optimized out>, expected=12, futex_word=0x5591072435e4) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 #1 futex_wait_simple (private=<optimized out>, expected=12, futex_word=0x5591072435e4) at ../sysdeps/nptl/futex-internal.h:135 #2 __pthread_cond_destroy (cond=0x5591072435c0) at pthread_cond_destroy.c:54 #3 0x00007fcf06b3bcee in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2629 #4 apr_pool_destroy (pool=0x5591071ac6b8) at memory/unix/apr_pools.c:1000 #5 0x00007fcf06b3bccd in apr_pool_destroy (pool=0x55910711a918) at memory/unix/apr_pools.c:997 #6 0x00007fcf01e86e55 in clean_child_exit (code=0) at worker.c:436 #7 0x00007fcf01e8795e in child_main (child_num_arg=child_num_arg@entry=5, child_bucket=child_bucket@entry=0) at worker.c:1197 #8 0x00007fcf01e87ffb in make_child (s=0x559107069650, slot=slot@entry=5, bucket=0) at worker.c:1328 #9 0x00007fcf01e8806d in startup_children (number_to_start=16, number_to_start@entry=20) at worker.c:1355 #10 0x00007fcf01e88c03 in server_main_loop (num_buckets=1, remaining_children_to_start=20) at worker.c:1669 #11 worker_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at worker.c:1761 #12 0x0000559105d346da in ap_run_mpm (pconf=0x5591070403a8, plog=0x55910706d5c8, s=0x559107069650) at mpm_common.c:94 #13 0x0000559105d2d01c in main (argc=<optimized out>, argv=<optimized out>) at main.c:819
On Thu, Jun 3, 2021 at 6:30 PM Eric Covener <cove...@gmail.com> wrote: > On Thu, Jun 3, 2021 at 6:23 AM Jay Ce <jceh12...@gmail.com> wrote: > > > > Hi, > > > > I noticed that sometimes when performing a graceful restart of httpd, > the CPU of the httpd process spikes and does not come down. This happens > more frequently if the web server is under load but can happen even when > not serving traffic. Has anyone else encountered this problem? And what can > I do to resolve this? > > > > I did not notice any error messages in the error log and only see the > following: > > Sat May 29 03:33:01.424748 2021] [mpm_worker:notice] [pid 2773:tid > 140527163445824] AH00297: SIGUSR1 received. Doing graceful restart > > AH00558: httpd: Could not reliably determine the server's fully > qualified domain name, using 127.0.0.1. Set the 'ServerName' directive > globally to suppress this message > > [Sat May 29 03:33:01.485368 2021] [http2:warn] [pid 2773:tid > 140527163445824] AH02951: mod_ssl does not seem to be enabled > > [Sat May 29 03:33:01.485953 2021] [mpm_worker:notice] [pid 2773:tid > 140527163445824] AH00292: Apache/2.4.46 () configured -- resuming normal > operations > > [Sat May 29 03:33:01.485964 2021] [core:notice] [pid 2773:tid > 140527163445824] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' > > [Sat May 29 03:33:01.486045 2021] [mpm_worker:warn] [pid 2773:tid > 140527163445824] AH00291: long lost child came home! (pid 25489) > > > > > > From performing a top > > top - 17:44:22 up 5 days, 16:04, 1 user, load average: 7.00, 7.13, 8.27 > > Threads: 384 total, 8 running, 316 sleeping, 0 stopped, 1 zombie > > %Cpu(s): 99.8 us, 0.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > > KiB Mem : 7807268 total, 4124256 free, 543316 used, 3139696 > buff/cache > > KiB Swap: 1310716 total, 1310716 free, 0 used. 6949164 avail > Mem > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 23677 apache 20 0 2293264 7524 4148 R 58.6 0.1 4582:47 httpd > > 23664 apache 20 0 2293264 7524 4148 R 58.2 0.1 4581:57 httpd > > 23636 apache 20 0 2293264 7524 4148 R 57.6 0.1 4582:16 httpd > > 23660 apache 20 0 2293264 7524 4148 R 57.2 0.1 4580:51 httpd > > 23698 apache 20 0 2293264 7524 4148 R 56.6 0.1 4582:17 httpd > > 23648 apache 20 0 2293264 7524 4148 R 55.9 0.1 4580:52 httpd > > 23668 apache 20 0 2293264 7524 4148 R 53.9 0.1 4581:23 httpd > > 1 root 20 0 128600 8592 5524 S 0.0 0.1 6:02.91 > systemd > > > > Can you get backtraces from a few such process with "pstack" or gdb? > > https://httpd.apache.org/dev/debugging.html#backtrace > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > >
