Thanks Lyn. It should have been obvious to me to expand the field display width! :)
Anyway, the full error was “launch_failed_requeued_held”. Using scontrol I saw that the affected jobs had a compute node in common, which in turn ended up having quite a few issues in its log files. So, that node is now in maintenance and we’ll see if the issue reappears. Many thanks, -- Ed Swindelles From: Lyn Gerner [mailto:[email protected]] Sent: Thursday, August 4, 2016 11:04 AM To: slurm-dev <[email protected]> Subject: [slurm-dev] Re: SLURM jobs pending due to "(launch" ? Hi Ed, Several Slurm goodies related to your question: Run 'squeue -j <jobID> -o "%13i %20j %55R" ' to show 13 chars of jobid, 20 chars of job name, and 55 chars of the reason field for a partcular job. Define SQUEUE_FORMAT in your env, with a customized output string such as the above, as described in the man page. You can also see the full reason field thru scontrol show job <jobID>. Regards, Lyn On Thu, Aug 4, 2016 at 4:15 AM, Swindelles, Ed <[email protected] <mailto:[email protected]> > wrote: Hi SLURM Community – In the last 2 weeks some of our users’ jobs have become stuck in a PENDING state forever, with stimes reporting a strange REASON_FOR_WAIT, simply “(launch”. The stimes command will also show a negative value for JOB_STARTS_IN (see screenshot below). This has only happened on a small fraction of our jobs, but often enough to create a few tickets. And, if we cancel the jobs and resubmit them, the problem almost always goes away. Has anyone seen this before? Our SLURM version is 15.08.11. The only error present in our SLURM log files is the following, from Slurmdbd.log: error: We have more time than is possible (14555124+7992096+0)(22547220) > 22422528 for cluster (this error is repeated quite frequently) Thank you, -- Ed Swindelles Manager of Advanced Computing University of Connecticut, School of Engineering 860-486-4522 <tel:860-486-4522>
smime.p7s
Description: S/MIME cryptographic signature
