Hi Ed, Several Slurm goodies related to your question:
Run 'squeue -j <jobID> -o "%13i %20j %55R" ' to show 13 chars of jobid, 20 chars of job name, and 55 chars of the reason field for a partcular job. Define SQUEUE_FORMAT in your env, with a customized output string such as the above, as described in the man page. You can also see the full reason field thru scontrol show job <jobID>. Regards, Lyn On Thu, Aug 4, 2016 at 4:15 AM, Swindelles, Ed <[email protected]> wrote: > Hi SLURM Community – > > > > In the last 2 weeks some of our users’ jobs have become stuck in a PENDING > state forever, with stimes reporting a strange REASON_FOR_WAIT, simply > “(launch”. The stimes command will also show a negative value for > JOB_STARTS_IN (see screenshot below). This has only happened on a small > fraction of our jobs, but often enough to create a few tickets. And, if we > cancel the jobs and resubmit them, the problem almost always goes away. Has > anyone seen this before? > > > > Our SLURM version is 15.08.11. The only error present in our SLURM log > files is the following, from Slurmdbd.log: > > error: We have more time than is possible (14555124+7992096+0)(22547220) > > 22422528 for cluster > > *(this error is repeated quite frequently)* > > > > > > Thank you, > > > > -- > > Ed Swindelles > > Manager of Advanced Computing > > University of Connecticut, School of Engineering > > 860-486-4522 > > >
