Hi, a question about handling of stdout in the PrologSlurmctld. Slurm
version is 15.08.7, OS version is Ubuntu 16.04.2 LTS.
I've unequivocally verified the prolog is running. However, writes to
stdout do not appear anywhere that I've found.
The prolog:
#!/bin/bash
/bin/echo "Hello, world, prolog here"
touch /tmp/foo
V=`lsof +f -ap $BASHPID -d 0,1,2 `
echo "$V" > /tmp/foo
exit 0
The output of the lsof command, e.g., the contents of /tmp/foo:
ubuntu@controller:/tmp$ more foo
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
lsof 2914 slurm 0r FIFO 0,10 0t0 21396 pipe
lsof 2914 slurm 1w FIFO 0,10 0t0 21396 pipe
lsof 2914 slurm 2u CHR 1,3 0t0 6 /dev/null
Output of the slurmctld (slurmctld -D -vvv). The prolog completion is
in bold font. Note the 'connection refused' messages.
lurmctld: debug2: Testing job time limits and checkpoints
slurmctld: debug2: sched: Processing RPC: REQUEST_RESOURCE_ALLOCATION
from uid=1000
slurmctld: debug2: found 1 usable nodes from config containing server
slurmctld: debug2: sched: JobId=81 allocated resources: NodeList=server1
slurmctld: sched: _slurm_rpc_allocate_resources JobId=81
NodeList=server1 usec=492
slurmctld: debug2: _slurm_rpc_job_ready(81)=2 usec=3
slurmctld: debug2: _slurm_rpc_job_ready(81)=2 usec=3
*slurmctld: debug2: prolog_slurmctld job 81 prolog completed*
slurmctld: debug2: _slurm_rpc_job_ready(81)=3 usec=3
slurmctld: debug: Configuration for job 81 complete
slurmctld: debug: laying out the 1 tasks on 1 hosts server1 dist 1
slurmctld: debug2: Processing RPC: REQUEST_COMPLETE_JOB_ALLOCATION
from uid=1000, JobId=81 rc=0
slurmctld: job_complete: JobID=81 State=0x1 NodeCnt=1 WEXITSTATUS 0
slurmctld: debug2: Spawning RPC agent for msg_type SRUN_JOB_COMPLETE
slurmctld: debug2: Spawning RPC agent for msg_type SRUN_JOB_COMPLETE
slurmctld: debug2: Spawning RPC agent for msg_type REQUEST_TERMINATE_JOB
slurmctld: job_complete: JobID=81 State=0x8003 NodeCnt=1 done
slurmctld: debug2: _slurm_rpc_complete_job_allocation: JobID=81
State=0x8003 NodeCnt=1
slurmctld: debug2: got 1 threads to send out
slurmctld: debug2: got 1 threads to send out
slurmctld: debug2: slurm_connect failed: Connection refused
*slurmctld: debug2: Error connecting slurm stream socket at
127.0.0.1:36461: Connection refused*
slurmctld: debug2: Tree head got back 0 looking for 1
slurmctld: debug2: got 1 threads to send out
slurmctld: debug2: slurm_connect failed: Connection refused
*slurmctld: debug2: Error connecting slurm stream socket at
127.0.0.1:38803: Connection refused*
ps output:
ubuntu@controller:/tmp$ ps ax | grep slurm
3040 pts/0 Sl 0:00 slurmctld -D -vvv
3057 pts/1 S+ 0:00 grep --color=auto slurm
ubuntu@controller:/tmp$
If I start slurmctld as the root user, I get two instances:
root@controller:/vagrant# ps ax | grep slurm
3378 pts/0 S 0:00 sudo slurmctld -D -vvv
3379 pts/0 Sl 0:00 slurmctld -D -vvv
3411 pts/0 S+ 0:00 grep slurm
However, this doesn't matter w/r/t prolog standard output and the output
of the lsof command above is the same.
munge is running and logging to syslog, but there is nothing in syslog
which appears broken or which appears when srun / sbatch are invoked.
netstat output, numeric, for listeners:
ubuntu@controller:/tmp$ netstat -l -n
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6817 0.0.0.0:* LISTEN
tcp6 0 0 :::22 :::* LISTEN
udp 0 0 0.0.0.0:68 0.0.0.0:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 16069
/run/user/1000/systemd/private
unix 2 [ ACC ] SEQPACKET LISTENING 7992 /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 7988
/run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 7993
/run/lvm/lvmpolld.socket
unix 2 [ ACC ] STREAM LISTENING 7998
/run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 8912
/run/lvm/lvmetad.socket
unix 2 [ ACC ] STREAM LISTENING 11693 /run/uuidd/request
unix 2 [ ACC ] STREAM LISTENING 11694
/var/lib/lxd/unix.socket
unix 2 [ ACC ] STREAM LISTENING 11695
/var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 11696 /run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 11697 /run/snapd.socket
unix 2 [ ACC ] STREAM LISTENING 11698
/run/snapd-snap.socket
unix 2 [ ACC ] STREAM LISTENING 11709
@ISCSIADM_ABSTRACT_NAMESPACE
unix 2 [ ACC ] STREAM LISTENING 13517
/var/run/munge/munge.socket.2
ubuntu@controller:/tmp$
/etc/hosts:
ubuntu@controller:/tmp$ cat /etc/hosts
127.0.0.1 controller controller
127.0.0.1 localhost
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
127.0.1.1 ubuntu-xenial ubuntu-xenial
10.10.10.3 controller
10.10.10.4 server1
Seems like a pipe forms stdout and stdin for the PrologSlurmctld, and
something isn't listening on the far end of the pipe. Hence no
output. Suggestions welcome. Things were working in with a much
earlier version of Slurm (e.g., -llnl Version 2).
Thanks
Tim
PS It would be great were the Prolog / Epilog guide or some man page to
address the subject of task prolog & epilog standard I/O. Even an
overview would be helpful. There are various posts about appending to
user job output, but that's not what we're looking for here.
