Is there a way for Slurm to detect when a user quota has been exceeded? We use XFS and when users are over the quota they will get a "Disk quota exceeded" message, e.g., when trying to scp or create a new file. However if they are not aware of this and try using a sbatch file, they don't receive any notification and the Slurm logs (which they don't have access to) simply say this:
[2020-08-04T09:12:35.001] _slurm_rpc_submit_batch_job: JobId=5495 InitPrio=4294900561 usec=1964 [2020-08-04T09:12:35.567] email msg to u...@myschool.edu: Slurm Job_id=5495 Name=M2_alltrans Began, Queued time 00:00:01 [2020-08-04T09:12:35.567] backfill: Started JobId=5495 in defq on node001 [2020-08-04T09:12:35.824] prolog_running_decr: Configuration for JobId=5495 is complete [2020-08-04T09:12:35.916] _job_complete: JobId=5495 WEXITSTATUS 1 [2020-08-04T09:12:35.916] email msg to u...@myschool.edu : Slurm Job_id=5495 Name=M2_alltrans Failed, Run time 00:00:00, FAILED, ExitCode 1 [2020-08-04T09:12:35.916] _job_complete: JobId=5495 done
