Package: gridengine-exec Version: 8.1.9+dfsg-4 Severity: important Tags: upstream patch
Dear Maintainer, We have an existing queue environment based on GridEngine and after upgrading our grid to Debian 9 "Stretch", attempts to "qlogin" would fail after the qlogin was assigned to a node. We have GridEngine configured to send the admins mail when this happens and I'll give an example email below that shows how it fails. It fails while trying to call seteuid(), I'll explain more below. This happens just after running the prolog script (note on the meaning of prolog: it's an administrator-specifiable script that runs just before jobs are started: qconf -sq all.q | grep prolog prolog /var/lib/gridengine/default/common/prolog.sh and note: I am not 100% sure that it even matters that we use a prolog, but I think it might be relevant and I suspect it might help explain why this problem hasn't happened to others). Anyway the real error seems to be of this type: "Cannot reset euid yshao due to Operation not permitted" The issue seems to be that the GridEngine code uses seteuid() to set the effective userid to root and then back to the job-submitter, and then directly back to 'sgeadmin' without going via root, and it seems there is some kind of constraint that you can't use seteuid() to change from one regular user to another, you have to go via root. "man seteuid()" is vague on this (speaking only of privilege without defining it precisely), but comments in the GridEngine code indicate there is such a constraint, and that code goes out of its way to first seteuid() to root before changing to the job-submitter, which is odd because when changing back from the job-submitter to sgeadmin it doesn't go via root. The intended properties of Linux in this regard, in terms of when exactly you're allowed to call seteuid() with a certain value, is very complicated and although I've read all kinds of man pages I've never found an unambiguous description. Anyway, when I downloaded the source package, changed the code of shepherd.c to change user back via root, compiled, and replaced the 'sge_shepherd' binary on all our exec nodes, the problem disappeared. For the key code changes see here https://github.com/gridengine/gridengine/issues/10 where I have posted the part of the original code that fails and a diff, but note, I also made other openssl-version-related changes, not shown, to fix compilation errors. The real mystery to me is why the behavior of seteuid() would have changed, since it seems a super stable part of Linux-- or if it didn't change, what else might have changed from Debian 8 to 9 so that this part of the code suddenly stopped working-- and why they would have gone to the trouble in the original code of changing users with root in between if it wasn't even necessary in practice at the time. Another mysterious observation: this problem only happened on nodes on which we installed Debian 9 from scratch, and not on those on which we did an incremental upgrade from Debian 8, even though the nodes had (or at least were supposed to have) the same configuration and packages. Also there is a separate issue that I'd appreciate a response on: does anyone maintain the GridEngine code? Who in the Debian organization at least maintains the package? I'm considering persuading some other GridEngine users to help me maintain a github repo for GridEngine. (I'm the leader of a large open source project "kaldi" already, so I'm familiar with the process). ==== email from GridEngine is below; search for 'euid' for the first error Job 8887005 caused action: none User = yshao Queue = [email protected] Start Time = <unknown> End Time = <unknown> failed in prolog: 04/30/2018 21:52:23 [115:2598]: exit_status of prolog = 143 Shepherd trace: 04/30/2018 21:52:23 [0:2598]: shepherd called with uid = 0, euid = 0 04/30/2018 21:52:23 [0:2598]: qlogin_daemon = builtin 04/30/2018 21:52:23 [115:2598]: starting up 8.1.9 04/30/2018 21:52:23 [115:2598]: setpgid(2598, 2598) returned 0 04/30/2018 21:52:23 [115:2598]: do_core_binding: "binding" parameter not found in config file 04/30/2018 21:52:23 [115:2598]: calling fork_pty() 04/30/2018 21:52:23 [115:2598]: parent: forked "prolog" with pid 2600 04/30/2018 21:52:23 [115:2598]: using signal delivery delay of 120 seconds 04/30/2018 21:52:23 [115:2598]: parent: prolog-pid: 2600 04/30/2018 21:52:23 [61161:2598]: Cannot reset euid yshao due to Operation not permitted 04/30/2018 21:52:23 [115:2600]: child: starting son(prolog, /var/lib/gridengine/default/common/prolog.sh, 0, 10000); 04/30/2018 21:52:23 [61161:2598]: now sending signal TERM to pid -2600 04/30/2018 21:52:23 [61161:2598]: Cannot reset euid yshao due to Operation not permitted 04/30/2018 21:52:23 [61161:2598]: now sending signal TERM to pid -2600 04/30/2018 21:52:23 [115:2598]: Poll received POLLHUP (Hang up). Unregister the FD. 04/30/2018 21:52:23 [115:2598]: wait3 returned 2600 (status: 15; WIFSIGNALED: 1, WIFEXITED: 0, WEXITSTATUS: 0) 04/30/2018 21:52:23 [115:2598]: prolog exited with exit status 0 04/30/2018 21:52:23 [115:2598]: reaped "prolog" with pid 2600 04/30/2018 21:52:23 [115:2598]: prolog exited due to signal 04/30/2018 21:52:23 [115:2598]: prolog signaled: 15 04/30/2018 21:52:23 [115:2598]: exit_status of prolog = 143 04/30/2018 21:52:23 [115:2598]: no epilog script to start 04/30/2018 21:52:23 [115:2598]: writing exit status to qrsh: 0 04/30/2018 21:52:23 [115:2598]: sending UNREGISTER_CTRL_MSG with exit_status = "0" 04/30/2018 21:52:23 [115:2598]: sending to host: <null> 04/30/2018 21:52:23 [115:2598]: comm_write_message returned: can't find handle 04/30/2018 21:52:23 [115:2598]: close_parent_loop: comm_write_message() returned 0 instead of 1!!! 04/30/2018 21:52:23 [115:2598]: waiting for UNREGISTER_RESPONSE_CTRL_MSG 04/30/2018 21:52:23 [115:2598]: No connection or problem while waiting for message: 1 04/30/2018 21:52:23 [115:2598]: parent: cl_com_ignore_timeouts 04/30/2018 21:52:23 [115:2598]: parent: error in comm_cleanup_lib(): 3 04/30/2018 21:52:23 [115:2598]: parent: leaving close_parent_loop() Shepherd error: 04/30/2018 21:52:23 [115:2598]: exit_status of prolog = 143 Shepherd pe_hostfile: c08.clsp.jhu.edu 1 [email protected] UNDEFINED === It probably doesn't matter, but here is our prolog script: #!/bin/bash function test_ok { if [ ! -z "$JOB_SCRIPT" ] && [ "$JOB_SCRIPT" != QLOGIN ] && [ "$JOB_SCRIPT" != QRLOGIN ]; then if [ ! -f "$JOB_SCRIPT" ]; then echo "$0: warning: no such file $JOB_SCRIPT, will wait" 1>&2 return 1; fi fi if [ ! -z "$SGE_STDERR_PATH" ]; then if [ ! -d "`dirname $SGE_STDERR_PATH`" ]; then echo "$0: warning: no such directory $JOB_SCRIPT, will wait." 1>&2 return 1; fi fi return 0; } if ! test_ok; then sleep 2; if ! test_ok; then sleep 4; if ! test_ok; then sleep 8; fi fi fi exit 0; -- System Information: Debian Release: 9.4 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-6-amd64 (SMP w/24 CPU cores) Locale: LANG=en_US, LC_CTYPE=en_US (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C), LANGUAGE=en_US (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C) Shell: /bin/sh linked to /bin/bash Init: systemd (via /run/systemd/system) Versions of packages gridengine-exec depends on: ii csh [c-shell] 20110502-2.2+b1 ii debconf [debconf-2.0] 1.5.61 ii gridengine-client 8.1.9+dfsg-4 ii gridengine-common 8.1.9+dfsg-4 ii libc6 2.24-11+deb9u3 ii libhwloc5 1.11.5-1 ii libjemalloc1 3.6.0-9.1 ii libmunge2 0.5.12-1+b1 ii libssl1.0.2 1.0.2l-2+deb9u3 ii lsb-base 9.20161125 ii procps 2:3.3.12-3 ii tcsh [c-shell] 6.20.00-7 gridengine-exec recommends no packages. Versions of packages gridengine-exec suggests: ii openssh-server 1:7.4p1-10+deb9u3 -- debconf information: * shared/gridenginemaster: deb8qmaster.clsp.jhu.edu * shared/gridengineconfig: true * shared/gridenginecell: default
