> Today I discovered that sbatch can also create job steps (albeit awkwardly, via --jobid), and it can obviously run sbatch scripts... but as far as I can tell, it cannot run synchronously!
Rats. I just discovered that sbatch has a "--wait" option, but it must have been added after 2.2.4, because it isn't available on this machine. Michael On Thu, Sep 20, 2018 at 7:01 PM Michael Lamparski <diagonaldev...@gmail.com> wrote: > Hello all, > > For years I've been looking for what I might consider to be the holy grail > of composable resource allocation in slurm jobs: > > * A command that can be run inside of an sbatch script... > * ...which immediately and synchronously invokes another sbatch script > (which may or may not invoke mpirun in turn)... > * ...using a subset of the currently allocated resources. > > This is the smallest unit of functionality that would compose well with > existing tools in UNIX for orchestration. For instance, I could use xargs > as a semaphore to let each node work on one input at a time, and for a > given input I could have an arbitrarily complex python script decide > dynamically what computations to run. > > Years of Google and manpage searches have continually failed me. > > * salloc can synchronously run an sbatch script, but as far as I can tell, > it cannot make job steps, only jobs. > * srun can run sychronously and make job steps, but as far as I can tell, > it cannot call a script which calls mpirun (it insists on *replacing* > mpirun) > * Today I discovered that sbatch can also create job steps (albeit > awkwardly, via --jobid), and it can obviously run sbatch scripts... but as > far as I can tell, it cannot run synchronously! > > One can't help but wonder whether this is a deliberate omission or just > criminal oversight! > > Today I snapped and started working on a synchronous wrapper around > sbatch; the plan is to use --jobid=$SLURM_JOB_ID, find out the job step id > (somehow...), and then sattach to it. I say this knowing it'll probably > make your skin crawl. And I ask: What do you think I ought to do instead? > > Michael >
