All,
I have several clusters that are all connected to a standalone slurmdb
server. They are not federated.
I can check the queues and do everything from that system using -M
<cluster> for most commands. However, if I try to get a shell within a
job (eg: srun -M cluster-a --pty bash), it queues up, but when it tries
to run, I get an error:
$ srun -M clustera -n16 --pty bash
srun: job 6 has been allocated resources
srun: error: Error connecting, bad data: family = 2, port = 0
srun: error: Task launch for 6.0 failed on node ip-0A312014:
Communication connection failure
srun: error: Application launch failed: Communication connection failure
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete
And in the slurmctld log for the cluster master:
Dec 17 19:50:04 nastran-master slurmctld[54739]: error:
slurm_receive_msg [10.49.32.20:44022]: Zero Bytes were transmitted or
received
Dec 17 19:50:07 nastran-master slurmctld[54739]: error:
slurm_receive_msg [10.49.32.20:44046]: Zero Bytes were transmitted or
received
Dec 17 19:50:08 nastran-master slurmctld[54739]: update_node: node
ip-0A312014 state set to DOWN
Dec 17 19:50:08 nastran-master slurmctld[54739]: Node ip-0A312014 now
responding
Dec 17 19:50:08 nastran-master slurmctld[54739]: node ip-0A312014
returned to service
Dec 17 19:50:09 nastran-master slurmctld[54739]: sched: Allocate JobId=6
NodeList=ip-0A312014 #CPUs=16 Partition=debug
Dec 17 19:50:09 nastran-master slurmctld[54739]: job_step_signal JobId=6
StepId=0 not found
Dec 17 19:50:41 nastran-master slurmctld[54739]: job_step_signal JobId=6
StepId=0 not found
Dec 17 19:50:41 nastran-master slurmctld[54739]: _job_complete: JobId=6
WTERMSIG 105
Dec 17 19:50:41 nastran-master slurmctld[54739]: _job_complete: JobId=6 done
Is this something that cannot be done from a system that is outside a
federated cluster?
Brian Andrus
