I am setting up slurm 17.02 with slurmdbd. Because I expect to have
multiple clusters sharing one slurmdbd node, I have followed the
instructions in the online documentation about how to use a separate munge
key for the connections to the slurmdbd.
That all works fine if I test with munge at the command line as the user
'slurm' under which the daemons run, e.g.:
munge -n --socket /var/run/munge/munge-accounting.socket.2 | ssh slurmdbd
unmunge --socket /var/run/munge/munge-accounting.socket.2
This is on Centos 7.3 so at each 'end' I have a modified systemd service
file that starts munged with the required extra options to use the right
socket, pidfile etc. The slurmctld node has two separate munge daemons,
one using the default socket and one using the one as shown above. Munge
is version 0.5-12 on both nodes, this was required in order to support the
multiple sockets.
The systemd startup file has this entry on both servers:
ExecStart=/usr/sbin/munged --socket /var/run/munge/munge-accounting.socket.2
--key-file /etc/munge/munge-accounting.key --pid-file /var/run/munge/munged-
accounting.pid
The munge user (a local account) is identical on both servers:
# id munge
uid=989(munge) gid=987(munge) groups=987(munge)
The slurm user is also identical, as it is an AD account.
I have set in the slurmdbd.conf file:
# Authentication info
AuthType=auth/munge
AuthInfo=*/var/run/munge/munge-accounting.socket.2*
And in the slurm.conf file on the slurmctld node:
AuthType=auth/munge
CryptoType=crypto/munge
AccountingStorageHost=slurmdbd.rothamsted.ac.uk
AccountingStoragePass=*/var/run/munge/munge-accounting.socket.2*
AccountingStoragePort=6819
AccountingStorageType=accounting_storage/slurmdbd
However when I start slurmctld it fails as it cannot talk to slurmdbd; the
problem is apparently with munge. This is the slurmctld logfile, I started
it in the foreground as user slurm with -vvv:
[2017-05-04T11:19:59.584] Not running as root. Can't drop supplementary
groups
[2017-05-04T11:19:59.589] Warning: Core limit is only 0 KB
[2017-05-04T11:19:59.590] slurmctld version 17.02.2 started on cluster
rothhpc2
[2017-05-04T11:19:59.597] Munge cryptographic signature plugin loaded
[2017-05-04T11:19:59.599] Consumable Resources (CR) Node Selection plugin
loaded with argument 20
[2017-05-04T11:19:59.602] preempt/none loaded
[2017-05-04T11:19:59.606] debug: Checkpoint plugin loaded: checkpoint/none
[2017-05-04T11:19:59.608] debug: AcctGatherEnergy NONE plugin loaded
[2017-05-04T11:19:59.611] debug: AcctGatherProfile NONE plugin loaded
[2017-05-04T11:19:59.614] debug: AcctGatherInfiniband NONE plugin loaded
[2017-05-04T11:19:59.616] debug: AcctGatherFilesystem NONE plugin loaded
[2017-05-04T11:19:59.617] debug2: No acct_gather.conf file
(/etc/slurm/acct_gather.conf)
[2017-05-04T11:19:59.620] debug: Job accounting gather LINUX plugin loaded
[2017-05-04T11:19:59.622] ExtSensors NONE plugin loaded
[2017-05-04T11:19:59.625] debug: switch NONE plugin loaded
[2017-05-04T11:19:59.626] debug: power_save module disabled, SuspendTime <
0
[2017-05-04T11:19:59.627] debug: No backup controller to shutdown
[2017-05-04T11:19:59.630] *Accounting storage SLURMDBD plugin loaded with
AuthInfo=/var/run/munge/munge-accounting.socket.2*
[2017-05-04T11:19:59.643] debug: Munge authentication plugin loaded
[2017-05-04T11:20:01.666] error: slurm_persist_conn_open: Something
happened with the receiving/processing of the persistent connection init
message to slurmdbd.rothamsted.ac.uk:6819: Failed to unpack Unknown message
[2017-05-04T11:20:01.666] error: slurmdbd: Sending PersistInit msg: No error
[2017-05-04T11:20:01.667] debug: Association database appears down,
reading from state file.
[2017-05-04T11:20:01.669] debug2: No association state file
(/var/spool/slurm/assoc_mgr_state) to recover
[2017-05-04T11:20:01.669] fatal: You are running with a database but for
some reason we have no TRES from it. This should only happen if the
database is down and you don't have any state files.
You can see that it has picked the right munge socket from the
configuration file. On the slurmdbd node however I see this is the
slurmdbd.log:
[2017-05-04T11:20:01.651] error: If munged is up, restart with
--num-threads=10
[2017-05-04T11:20:01.651] error: Munge decode failed: *Failed to access
"/var/run/munge/munge.socket.2"*: No such file or directory
[2017-05-04T11:20:01.651] error: slurm_unpack_received_msg:
REQUEST_PERSIST_INIT has authentication error: Socket communication error
[2017-05-04T11:20:01.651] error: slurm_unpack_received_msg: Protocol
authentication error
[2017-05-04T11:20:01.662] error: CONN:11 Failed to unpack Unknown message
[2017-05-04T11:20:03.675] error: If munged is up, restart with
--num-threads=10
[2017-05-04T11:20:03.675] error: Munge decode failed: Failed to access
"/var/run/munge/munge.socket.2": No such file or directory
[2017-05-04T11:20:03.676] error: slurm_unpack_received_msg:
REQUEST_PERSIST_INIT has authentication error: Socket communication error
[2017-05-04T11:20:03.676] error: slurm_unpack_received_msg: Protocol
authentication error
[2017-05-04T11:20:03.686] error: CONN:10 Failed to unpack Unknown message
It is trying to access the default munge socket, which does not exist, even
though the slurmdbd.conf file tells it which socket to use. If I run the
slurmctld under strace I can see a reference to that (wrong) socket:
stat("/home/apps/slurm/lib64/slurm/auth_munge.so", {st_mode=S_IFREG|0755,
st_size=27416, ...}) = 0
access("/home/apps/slurm/lib64/slurm/auth_munge.so", R_OK) = 0
open("/home/apps/slurm/lib64/slurm/auth_munge.so", O_RDONLY|O_CLOEXEC) = 7
read(7, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\23\0\0\0\0\0\0"...,
832) = 832
fstat(7, {st_mode=S_IFREG|0755, st_size=27416, ...}) = 0
mmap(NULL, 2109832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) =
0x7f79f5663000
mprotect(0x7f79f5666000, 2093056, PROT_NONE) = 0
mmap(0x7f79f5865000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE,
7, 0x2000) = 0x7f79f5865000
close(7) = 0
mprotect(0x7f79f5865000, 4096, PROT_READ) = 0
poll([{fd=2, events=POLLOUT}], 1, 5000) = 1 ([{fd=2, revents=POLLOUT}])
fstat(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(2, "slurmctld: debug: Munge authent"..., 54slurmctld: debug: Munge
authentication plugin loaded
) = 54
poll([{fd=4, events=POLLOUT}], 1, 5000) = 1 ([{fd=4, revents=POLLOUT}])
fstat(4, {st_mode=S_IFREG|0600, st_size=1915, ...}) = 0
write(4, "[2017-05-04T11:19:59.643] debug:"..., 69) = 69
rt_sigaction(SIGALRM, {SIG_DFL, , SA_RESTORER, 0x7f79f88d0370}, {SIG_DFL,
[], 0}, 8) = 0
*stat("/var/run/munge/munge.socket.2*", {st_mode=S_IFSOCK|0777, st_size=0,
...}) = 0
socket(PF_LOCAL, SOCK_STREAM, 0) = 7
fcntl(7, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(7, {sa_family=AF_LOCAL, sun_path="/var/run/munge/munge.socket.2"},
110) = 0
writev(7, [{"\0`mK\4\2\0\0\0\0\24", 11}, {"\1\1\1\0\0\0\0\0\377\377\
377\377\377\377\377\377\0\0\0\0", 20}], 2) = 31
read(7, 0x7ffdadc46e70, 11) = -1 EAGAIN (Resource temporarily
unavailable)
poll([{fd=7, events=POLLIN}], 1, 3000) = 1 ([{fd=7,
revents=POLLIN|POLLHUP}])
read(7, "\0`mK\4\3\0\0\0\0\206", 11) = 11
read(7, "\0\0\0\0\0\200MUNGE:AwQDAAD1sgHkaqRU4gzQ"..., 134) = 134
close(7) = 0
rt_sigaction(SIGALRM, {SIG_DFL, , SA_RESTORER, 0x7f79f88d0370}, {SIG_DFL, ,
SA_RESTORER, 0x7f79f88d0370}, 8) = 0
brk(0) = 0x2825000
brk(0x2846000) = 0x2846000
rt_sigaction(SIGPIPE, {SIG_IGN, , SA_RESTORER, 0x7f79f88d0370}, {SIG_DFL,
[], 0}, 8) = 0
fcntl(6, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(6, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
poll([{fd=6, events=POLLOUT}], 1, 10000) = 1 ([{fd=6, revents=POLLOUT}])
recvfrom(6, 0x7ffdadc46f60, 1, 0, 0, 0) = -1 EAGAIN (Resource temporarily
unavailable)
sendto(6, "\0\0\0\300", 4, 0, NULL, 0) = 4
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
fcntl(6, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(6, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
poll([{fd=6, events=POLLOUT}], 1, 10000) = 1 ([{fd=6, revents=POLLOUT}])
recvfrom(6, 0x7ffdadc46f60, 1, 0, 0, 0) = -1 EAGAIN (Resource temporarily
unavailable)
sendto(6, "\31d\37\0\0\2\0\0\31d\0\0\0\21\0\0\0\0\0\0\0\0\0\0\0\0\0\vauth"...,
192, 0, NULL, 0) = 192
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
rt_sigaction(SIGPIPE, {SIG_DFL, , SA_RESTORER, 0x7f79f88d0370}, {SIG_IGN, ,
SA_RESTORER, 0x7f79f88d0370}, 8) = 0
brk(0) = 0x2846000
brk(0) = 0x2846000
brk(0x2842000) = 0x2842000
brk(0) = 0x2842000
poll([{fd=6, events=POLLIN}], 1, 45000) = 1 ([{fd=6, revents=POLLIN}])
read(6, "\0\0\0-", 4) = 4
poll([{fd=6, events=POLLIN}], 1, 45000) = 1 ([{fd=6, revents=POLLIN}])
read(6, "\5\231\0\0\0!Failed to unpack Unknown m"..., 45) = 45
poll([{fd=2, events=POLLOUT}], 1, 5000) = 1 ([{fd=2, revents=POLLOUT}])
fstat(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(2, "slurmctld: error: slurm_persist_"..., 202slurmctld: error:
slurm_persist_conn_open: Something happened with the receiving/processing
of the persistent connection init message to slurmdbd.rothamsted.ac.uk:6819:
Failed to unpack Unknown message
I am unclear why the slurmctld is even contemplating using the default
munge socket - it *does* exist on the slurmctld node but is the wrong
daemon - it is for communication to the compute nodes and uses a different
key.
The end result seems to be that either the wrong key is being used for the
link to the slurmdbd, or slurmctld is in some way telling slurmdbd which
socket to use, but it doesn't exist. The configuration files seem to
follow the instructions. All servers use the same slurm.conf (it is
counted to all of them from a central copy).
I should also add that I did try with the slurmdbd node using the munge
default configuration. I then do not see the issue about the socket not
being found but the munge authentication still fails, as if slurmctld has
used the wrong munge socket (and daemon, and key).
Has anyone else seen this behaviour or managed to configured the
multiple-munge domains successfully ?
William
