Hi David, I would recommend using one of the many workflow management systems (WMS) that can define job dependencies programmatically rather than fuss with base Slurm commands. Some examples I have used or have been recommended to me:
- https://pegasus.isi.edu/ - https://cylc.github.io/ - http://parsl-project.org/ - https://airflow.apache.org/ Each of these runs in production on a number of major projects and has active support from developers. Making a pipeline DAG makes managing jobs and rerunning tests so much nicer. No reason to write from scratch what already exists. Cheers, Matt West ________________________________ From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of David Henkemeyer <david.henkeme...@gmail.com> Sent: Monday, May 9, 2022, 22:59 To: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: [slurm-users] How to run a job at the end of a set of jobs CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Prologue is a feature whereby I can run something after a single job. Is there a best practice for running a job after a set of jobs? We submit a bunch of jobs to a bunch of nodes, and after all the jobs are done, we would like to submit a "utility job" on each node, but it has to be the last job. What would be the various ways to achieve this? Thanks David
