Hi Lucas, This old thread might help:
https://groups.google.com/forum/#!topic/slurm-devel/TQcerLLEKAU Paddy On Fri, Jan 20, 2017 at 10:00:00AM -0800, Lucas Vuotto wrote: > > Hi all, > sreport was showing that an user was using more CPU hours per week > than available. After checking the output of sacct, we found that some > jobs from an array didn't ended: > > $ sacct -j 69204 -o jobid%-14,state%6,start,elapsed,end > > JobID State Start Elapsed End > > -------------- ------ ------------------- ---------- ------------------- > 69204_[1-1000] FAILED 2016-11-09T17:46:50 00:00:00 2016-11-09T17:46:50 > 69204_1 FAILED 2016-11-09T17:46:44 71-20:25:55 Unknown > 69204_2 FAILED 2016-11-09T17:46:44 71-20:25:55 Unknown > [...] > 69204_295 FAILED 2016-11-09T17:46:46 71-20:25:53 Unknown > 69204_296 FAILED 2016-11-09T17:46:46 71-20:25:53 Unknown > 69204_297 FAILED 2016-11-09T17:46:46 00:00:00 2016-11-09T17:46:46 > [...] > 69204_999 FAILED 2016-11-09T17:46:50 00:00:00 2016-11-09T17:46:50 > > It seems that somehow those jobs got stucked (~72 days after > 2016-11-09 is today, 2017-01-20, and that's why the wrong reports). > scancel says that 69204 is an invalid job id. > > Any idea on how to fix this? We're thinking about deleting the entries > of those jobs in the DB. Is it safe to run "arbitrary" commands in the > DB, bypassing slurmdbd? > > Thanks in advance. > > > -- lv. > > > -- lv. > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
