Just to add, squeue returns
JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON)
7 standard hostname ubuntu R 0:21 1 node0
James Venning
On 31 August 2016 at 12:57, James Andrew Venning <[email protected]> wrote:
> Hi all again,
> Initial problems are fixed, but there are more - thanks in advance for
> reading this!
>
> Ubuntu 14.04 with Slurm 2.6.5-1
>
> Master and controller seem to be talking to each other:
> scontrol ping
> Slurmctld(primary/backup) at slurm-master/(NULL) are UP/DOWN
> exact same message from each slave
>
> ON one of the slaves, node0, I start the daemon with
> sudo slurmd -Dvvv
>
> which returns
> slurmd: topology NONE plugin loaded
> slurmd: CPU frequency setting not configured for this node
> slurmd: task NONE plugin loaded
> slurmd: auth plugin for Munge (http://code.google.com/p/munge/) loaded
> slurmd: debug: spank: opening plugin stack /etc/slurm-llnl/plugstack.conf
> slurmd: Munge cryptographic signature plugin loaded
> slurmd: Warning: Core limit is only 0 KB
> slurmd: slurmd version 2.6.5 started
> slurmd: Job accounting gather NOT_INVOKED plugin loaded
> slurmd: switch NONE plugin loaded
> slurmd: slurmd started on Wed, 31 Aug 2016 02:48:03 +0000
> slurmd: CPUs=16 Boards=1 Sockets=16 Cores=1 Threads=1 Memory=64431
> TmpDisk=10042 Uptime=77963
> slurmd: AcctGatherEnergy NONE plugin loaded
> slurmd: AcctGatherProfile NONE plugin loaded
> slurmd: AcctGatherInfiniband NONE plugin loaded
> slurmd: AcctGatherFilesystem NONE plugin loaded
> slurmd: debug2: No acct_gather.conf file (/etc/slurm-llnl/acct_gather.
> conf)
>
> Which seems fine?
>
> sinfo on any machine returns
> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
> standard* up infinite 1 down* node2
> standard* up infinite 2 idle node[0-1]
>
> node2 isn't connected yet, so I'm not concerned about that.
>
> When I run a job, srun -N1 /bin/hostname
>
> The master debug says:
> slurmctld: debug2: sched: Processing RPC: REQUEST_RESOURCE_ALLOCATION from
> uid=1000
> slurmctld: debug: job_submit_plugin_submit: usec=1
> slurmctld: debug2: found 3 usable nodes from config containing node[0-2]
> slurmctld: debug2: sched: JobId=7 allocated resources: NodeList=node0
> slurmctld: sched: _slurm_rpc_allocate_resources JobId=7 NodeList=node0
> usec=3578
> slurmctld: debug2: _slurm_rpc_job_ready(7)=3 usec=14
> slurmctld: debug2: Processing RPC: REQUEST_JOB_STEP_CREATE from uid=1000
> slurmctld: debug: Configuration for job 7 complete
> slurmctld: debug: laying out the 1 tasks on 1 hosts node0 dist 1
> slurmctld: sched: _slurm_rpc_job_step_create: StepId=7.0 node0 usec=6717
>
> And the node's says:
> slurmd: debug2: got this type of message 1008
> slurmd: debug2: got this type of message 6001
> slurmd: debug2: Processing RPC: REQUEST_LAUNCH_TASKS
> slurmd: debug: task_slurmd_launch_request: 7.0 0
> slurmd: launch task 7.0 request from [email protected] (port 12449)
> slurmd: debug: Checking credential with 256 bytes of sig data
> slurmd: debug: Calling /usr/sbin/slurmstepd spank prolog
> spank-prolog: Reading slurm.conf file: /etc/slurm-llnl/slurm.conf
> spank-prolog: Running spank/prolog for jobid [7] uid [1000]
> spank-prolog: spank: opening plugin stack /etc/slurm-llnl/plugstack.conf
> slurmd: debug: task_slurmd_reserve_resources: 7 0
>
> The sinfo changes to allocated:
> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
> standard* up infinite 1 down* node2
> standard* up infinite 1 alloc node0
> standard* up infinite 1 idle node1
>
> And scontrol show job 7 returns:
> JobId=7 Name=hostname
> UserId=ubuntu(1000) GroupId=ubuntu(1000)
> Priority=4294901754 Account=(null) QOS=(null)
> JobState=RUNNING Reason=None Dependency=(null)
> Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
> RunTime=00:01:29 TimeLimit=UNLIMITED TimeMin=N/A
> SubmitTime=2016-08-31T02:51:06 EligibleTime=2016-08-31T02:51:06
> StartTime=2016-08-31T02:51:06 EndTime=Unknown
> PreemptTime=None SuspendTime=None SecsPreSuspend=0
> Partition=standard AllocNode:Sid=slurm-master:3410
> ReqNodeList=(null) ExcNodeList=(null)
> NodeList=node0
> BatchHost=node0
> NumNodes=1 NumCPUs=16 CPUs/Task=1 ReqS:C:T=*:*:*
> MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
> Features=(null) Gres=(null) Reservation=(null)
> Shared=0 Contiguous=0 Licenses=(null) Network=(null)
> Command=/bin/hostname
> WorkDir=/home/ubuntu
>
> But the job doesn't run... Any help is appreciated greatly.
>
> James Venning
>
