A few more things to note: - (Should have mentioned this earlier) running Slurm 17.11.7 ( via https://launchpad.net/~jonathonf/+archive/ubuntu/slurm ) - Restarted slurmctld and slurmdbd, but still getting the slurmdbd errors as before in slurmctld.log - Ran "mysqlcheck --databases slurm_acct_db --auto-repair", output was: slurm_acct_db.acct_coord_table OK slurm_acct_db.acct_table OK slurm_acct_db.clus_res_table OK slurm_acct_db.cluster_table OK slurm_acct_db.convert_version_table OK slurm_acct_db.federation_table OK slurm_acct_db.macluster_assoc_table OK slurm_acct_db.macluster_assoc_usage_day_table OK slurm_acct_db.macluster_assoc_usage_hour_table OK slurm_acct_db.macluster_assoc_usage_month_table OK slurm_acct_db.macluster_event_table OK slurm_acct_db.macluster_job_table OK slurm_acct_db.macluster_last_ran_table OK slurm_acct_db.macluster_resv_table OK slurm_acct_db.macluster_step_table OK slurm_acct_db.macluster_suspend_table OK slurm_acct_db.macluster_usage_day_table OK slurm_acct_db.macluster_usage_hour_table OK slurm_acct_db.macluster_usage_month_table OK slurm_acct_db.macluster_wckey_table OK slurm_acct_db.macluster_wckey_usage_day_table OK slurm_acct_db.macluster_wckey_usage_hour_table OK slurm_acct_db.macluster_wckey_usage_month_table OK slurm_acct_db.qos_table OK slurm_acct_db.res_table OK slurm_acct_db.table_defs_table OK slurm_acct_db.tres_table OK slurm_acct_db.txn_table OK slurm_acct_db.user_table OK - Nothing in /var/log/mysql/error.log for as far back as logs go - Ran "sacctmgr show runaway", there were a LOT of runaway jobs; chose "Y" to fix, then output of "sacctmgr show runaway" was nil. A few minutes later however, "sacctmgr show runaway" had entries again.
If someone knows what else I might try to isolate/resolve this issue, please kindly assist... From: Will Dennis Sent: Tuesday, July 16, 2019 2:43 PM To: slurm-users@lists.schedmd.com Subject: sacct issue: jobs staying in "RUNNING" state Hi all, Was looking at the running jobs on one groups cluster, and saw there was an insane amount of "running" jobs when I did a sacct -X -s R; then looked at output of squeue, and found a much more reasonable number... root@slurm-controller1:/ # sacct -X -p -s R | wc -l 8895 root@ slurm-controller1:/ # squeue | wc -l 43 In looking for the cause, I see a large amount of the following in the slurmctld.log file: [2019-07-16T09:36:51.464] error: slurmdbd: agent queue is full (20140), discarding DBD_STEP_START:1442 request [2019-07-16T09:40:27.515] error: slurmdbd: agent queue filling (20140), RESTART SLURMDBD NOW [2019-07-16T09:40:27.515] error: slurmdbd: agent queue is full (20140), discarding DBD_JOB_COMPLETE:1424 request [2019-07-16T09:40:27.515] error: slurmdbd: agent queue is full (20140), discarding DBD_STEP_COMPLETE:1441 request [2019-07-16T09:42:40.766] error: slurmdbd: agent queue filling (20140), RESTART SLURMDBD NOW [2019-07-16T09:42:40.766] error: slurmdbd: agent queue is full (20140), discarding DBD_STEP_START:1442 request [2019-07-16T09:46:05.905] error: slurmdbd: agent queue filling (20140), RESTART SLURMDBD NOW [2019-07-16T09:46:05.905] error: slurmdbd: agent queue is full (20140), discarding DBD_STEP_COMPLETE:1441 request [2019-07-16T09:46:05.905] error: slurmdbd: agent queue is full (20140), discarding DBD_JOB_COMPLETE:1424 request [2019-07-16T09:48:42.616] error: slurmdbd: agent queue filling (20140), RESTART SLURMDBD NOW [2019-07-16T09:48:42.616] error: slurmdbd: agent queue is full (20140), discarding DBD_JOB_COMPLETE:1424 request [2019-07-16T09:48:42.616] error: slurmdbd: agent queue is full (20140), discarding DBD_STEP_COMPLETE:1441 request [2019-07-16T09:53:00.188] error: slurmdbd: agent queue filling (20140), RESTART SLURMDBD NOW [2019-07-16T09:53:00.188] error: slurmdbd: agent queue is full (20140), discarding DBD_JOB_COMPLETE:1424 request [2019-07-16T09:53:00.189] error: slurmdbd: agent queue is full (20140), discarding DBD_STEP_COMPLETE:1441 request What may be the cause of this issue? And, is there any way now to correct the accounting records in the db? Thanks, Will
