Control: tags -1 - moreinfo Hi Axel,
On 2021-08-01 22:46:57 +0200, Axel Beckert wrote: > I've seen this zsh zombies at least Buster, too. See > http://savannah.gnu.org/bugs/?func=detailitem&item_id=25089 > > Could your issue be related or even identical to the one above in the > upstream bug tracker? (I currently assume so, hence the "confirmed" > tag.) There aren't many details, but this seems to be the same bug: "Maybe also worth to note is that these zombies process are not just there for a few minutes and then get cleaned up. Most of them (but not all) stay for at least days, if not until I exit the screen session." which is more or less what I observe (I confirm the "Most of them (but not all)"). > Unfortunately so far I wasn't able to reproduce this on demand. Currently I can reproduce the bug all the time on this VM when starting new screen sessions. Note: I haven't rebooted the VM yet (in case this is related to some kernel status). > And for some reason, so far this issue — if seen at all — has mostly > been seen when Screen runs on virtual machines. The reason for this is > though unclear, at least to me. I would say that this could have an influence on the behavior in the case of a race condition. Comment #12 says: "The reason for this is that in the C language only some very narrow set of library functions can be invoked from the signal handler" However, the signal handler in screen.c is named SigChld and does static sigret_t SigChld SIGDEFARG { debug("SigChld()\n"); GotSigChld = 1; SIGRETURN; } which is OK (the function named SigChldHandler is actually the callback function, not the signal handler), possibly except the "debug" (but I don't think it does anything without debugging). Note that GotSigChld is declared as static int GotSigChld; which is incorrect, AFAIK. It must be declared as volatile. Without that, there is a risk that the loop while (GotSigChld) { GotSigChld = 0; DoWait(); #ifdef SYSVSIGS signal(SIGCHLD, SigChld); #endif } be transformed into an "if" (itself transformed into a true condition, due to the context, but this second transformation doesn't matter). Now, if I understand correctly, the loop would matter only if two SIGCHLD are received in a short time, which probably doesn't occur in my case. Well... actually it does. See below. I've tried to see what happens with "strace -p PID" on the SCREEN process, but that's rather useless: While I don't use strace, I get zombies, but as soon as I use strace and quit the shell with Ctrl-D, all the zombies disappear. This could confirm that the cause is a race condition, as strace slows things down. In case this is useful, here's the strace output with 5 zombies, then strace -p PID, then quit the shell in the current screen window, then interrupted strace: select(1024, [3 4 5 6], [], NULL, {tv_sec=58, tv_usec=982211}) = 1 (in [3], left {tv_sec=47, tv_usec=980823}) read(3, "\4", 4096) = 1 gettimeofday({tv_sec=1627860672, tv_usec=147052}, NULL) = 0 select(1024, [3 4 5 6], [6], NULL, {tv_sec=47, tv_usec=952948}) = 1 (out [6], left {tv_sec=47, tv_usec=952939}) write(6, "\4", 1) = 1 gettimeofday({tv_sec=1627860672, tv_usec=163469}, NULL) = 0 select(1024, [3 4 5 6], [], NULL, {tv_sec=47, tv_usec=936531}) = 1 (in [6], left {tv_sec=47, tv_usec=936522}) read(6, "\0\33[?2004l\r\r\n", 4096) = 12 gettimeofday({tv_sec=1627860672, tv_usec=174073}, NULL) = 0 select(1024, [3 4 5 6], [3], NULL, {tv_sec=47, tv_usec=925927}) = 1 (out [3], left {tv_sec=47, tv_usec=925919}) write(3, "\r\n", 2) = 2 gettimeofday({tv_sec=1627860672, tv_usec=184401}, NULL) = 0 select(1024, [3 4 5 6], [], NULL, {tv_sec=47, tv_usec=915599}) = 1 (in [6], left {tv_sec=47, tv_usec=740073}) read(6, "\0\33[96mzlogout r139201\33(B\33[m\r\n", 4096) = 29 gettimeofday({tv_sec=1627860672, tv_usec=373145}, NULL) = 0 select(1024, [3 4 5 6], [3], NULL, {tv_sec=47, tv_usec=726855}) = 1 (out [3], left {tv_sec=47, tv_usec=726846}) write(3, "\33[27m\33[36m\33[49m\33[96mzlogout r139"..., 42) = 42 gettimeofday({tv_sec=1627860672, tv_usec=385146}, NULL) = 0 select(1024, [3 4 5 6], [], NULL, {tv_sec=47, tv_usec=714854}) = 1 (in [6], left {tv_sec=47, tv_usec=704494}) --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=25280, si_uid=1000, si_status=0, si_utime=12, si_stime=7} --- rt_sigreturn({mask=[]}) = 1 read(6, 0x7fff26608bc0, 4096) = -1 EIO (Input/output error) wait4(25280, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 25280 ioctl(3, TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0 ioctl(3, SNDCTL_TMR_START or TCSETS, {B38400 opost -isig -icanon -echo ...}) = 0 ioctl(3, TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0 gettimeofday({tv_sec=1627860672, tv_usec=423366}, NULL) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2971, ...}) = 0 gettimeofday({tv_sec=1627860672, tv_usec=430229}, NULL) = 0 gettimeofday({tv_sec=1627860672, tv_usec=433595}, NULL) = 0 ioctl(3, TCXONC, TCOON) = 0 ioctl(3, TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0 ioctl(3, SNDCTL_TMR_START or TCSETS, {B38400 opost -isig -icanon -echo ...}) = 0 ioctl(3, TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0 lseek(12, 0, SEEK_SET) = 0 alarm(0) = 0 rt_sigaction(SIGALRM, {sa_handler=0x7f2c90100e30, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f2c90006840}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f2c90006840}, 8) = 0 alarm(10) = 0 fcntl(12, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 read(12, "\2\0\0\0\0\0\0\0~\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\6\0\0\0l\2\0\0tty1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\6\0\0\0n\2\0\0hvc0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\1\0\0\0005\0\0\0~\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0&U\0\0pts/0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0JT\0\0pts/1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0\322A\0\0pts/2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\10\0\0\0\337U\0\0pts/3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\10\0\0\0\337U\0\0pts/4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\10\0\0\0\337U\0\0pts/5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0\233:\0\0pts/6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\10\0\0\0\337U\0\0pts/7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0Sb\0\0pts/8\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0\250U\0\0pts/9\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0SZ\0\0pts/10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 read(12, "\7\0\0\0Sb\0\0pts/11\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 384) = 384 fcntl(12, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 alarm(0) = 10 rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f2c90006840}, NULL, 8) = 0 rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f2c90006840}, {sa_handler=0x55c210b15420, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f2c90006840}, 8) = 0 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f2c8ff92a10) = 26065 wait4(26065, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 26065 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26065, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- rt_sigaction(SIGCHLD, {sa_handler=0x55c210b15420, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f2c90006840}, NULL, 8) = 0 chmod("/dev/pts/11", 0666) = 0 chown("/dev/pts/11", 0, 0) = -1 EPERM (Operation not permitted) close(6) = 0 munmap(0x7f2c8fb7e000, 290816) = 0 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 25388 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 25496 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 25612 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 25721 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 25829 wait4(-1, 0x7fff26609b2c, WNOHANG|WSTOPPED, NULL) = 0 stat("/srv/d_joooj/home/vinc17/.screen/25171.pts-0.joooj", {st_mode=S_IFSOCK|0700, st_size=0, ...}) = 0 gettimeofday({tv_sec=1627860672, tv_usec=814750}, NULL) = 0 select(1024, [3 4 5], [3], NULL, {tv_sec=47, tv_usec=285250}) = 1 (out [3], left {tv_sec=47, tv_usec=285237}) write(3, "\33[H\33[2J\33]2;vinc17@joooj - ~ | pt"..., 256) = 256 gettimeofday({tv_sec=1627860672, tv_usec=825728}, NULL) = 0 select(1024, [3 4 5], [3], NULL, {tv_sec=47, tv_usec=274272}) = 1 (out [3], left {tv_sec=47, tv_usec=274265}) write(3, "cluded: zfilesfct r125411 / zali"..., 256) = 256 gettimeofday({tv_sec=1627860672, tv_usec=836749}, NULL) = 0 select(1024, [3 4 5], [3], NULL, {tv_sec=47, tv_usec=263251}) = 1 (out [3], left {tv_sec=47, tv_usec=263244}) write(3, "ABLED\7\33]2;[0] vinc17@joooj - ~ |"..., 53) = 53 gettimeofday({tv_sec=1627860672, tv_usec=848449}, NULL) = 0 select(1024, [3 4 5], [], NULL, {tv_sec=47, tv_usec=251551} <detached ...> There are 8 wait4: * The first one (25280) corresponds to the shell process that was running in the window. * The second one (26065) corresponds to the clone that occurred when I quite the shell (what is this?). * The next 5 correspond to the 5 zombies. * The last one returns 0, meaning that there are no longer any remaining child processes. I'm wondering whether the issue is due to what happens with the clone. I suspect that there is a wait4 that catches this clone, but not the terminated shell process, hence the corresponding zombie. > Vincent Lefevre wrote: > > It seems more or less reproducbile: > > That would be good news! > > > 21983 SCREEN -c d_joooj/home/vinc17/.screenrc-mutt -T > > screen-256color-bce-s -dR > > Would you mind providing that .screenrc-mutt? This doens't matter: the problem is reproducible with a normal "screen", even if I remove the .screenrc file. > And now easy or how long does it take until zombies appear? I don't understand what you mean: as soon as I quit a shell running from a screen window, I get a zombie (same PID). > > But it may happen that zombies disappear. > > That's what I noticed, too, but it can take days, maybe weeks. Due to other SIGCHLDs from another terminated process started by SCREEN, but for some reason (more loaded machine / slower I/O?), things occur differently, just like with strace. -- Vincent Lefèvre <[email protected]> - Web: <https://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
