Hi SLURM Community -
In the last 2 weeks some of our users' jobs have become stuck in a PENDING state forever, with stimes reporting a strange REASON_FOR_WAIT, simply "(launch". The stimes command will also show a negative value for JOB_STARTS_IN (see screenshot below). This has only happened on a small fraction of our jobs, but often enough to create a few tickets. And, if we cancel the jobs and resubmit them, the problem almost always goes away. Has anyone seen this before? Our SLURM version is 15.08.11. The only error present in our SLURM log files is the following, from Slurmdbd.log: error: We have more time than is possible (14555124+7992096+0)(22547220) > 22422528 for cluster (this error is repeated quite frequently) Thank you, -- Ed Swindelles Manager of Advanced Computing University of Connecticut, School of Engineering 860-486-4522
