If the 500 parameters happened to be filenames, you could do adapt like
(appropriated from somewhere else, but I can’t find the reference quickly:
=====
#!/bin/bash
# get count of files in this directory
NUMFILES=$(ls -1 *.inp | wc -l)
# subtract 1 as we have to use zero-based indexing (first element is 0)
ZBNUMFILES=$(($NUMFILES - 1))
# submit array of jobs to SLURM
if [ $ZBNUMFILES -ge 0 ]; then
sbatch --array=0-$ZBNUMFILES array_job.sh
else
echo "No jobs to submit, since no input files in this directory.”
fi
=====
with:
=====
#!/bin/bash
#SBATCH --nodes=1 --ntasks-per-node=1 --cpus-per-task=1
#SBATCH --time=00:01:00
#SBATCH --job-name array_demo_2
echo "All jobs in this array have:"
echo "- SLURM_ARRAY_JOB_ID=${SLURM_ARRAY_JOB_ID}"
echo "- SLURM_ARRAY_TASK_COUNT=${SLURM_ARRAY_TASK_COUNT}"
echo "- SLURM_ARRAY_TASK_MIN=${SLURM_ARRAY_TASK_MIN}"
echo "- SLURM_ARRAY_TASK_MAX=${SLURM_ARRAY_TASK_MAX}"
echo "This job in the array has:"
echo "- SLURM_JOB_ID=${SLURM_JOB_ID}"
echo "- SLURM_ARRAY_TASK_ID=${SLURM_ARRAY_TASK_ID}"
# grab our filename from a directory listing
FILES=($(ls -1 *.inp))
FILENAME=${FILES[$SLURM_ARRAY_TASK_ID]}
echo "My input file is ${FILENAME}”
# make new directory, change into it, and run
mkdir ${FILENAME}_out
cd ${FILENAME}_out
echo "First 10 lines of ../${FILENAME} are:" > ${FILENAME}_results.out
head ../${FILENAME} >> ${FILENAME}_results.out
=====
If the 500 parameters were lines in a file, the same logic would apply:
- subtract 1 from the number of lines in the file to determine the array limit
- add 1 to ${SLURM_ARRAY_TASK_ID} to get a line number for a specific parameter
- something like "sed -n ‘${TASK_ID_PLUS_ONE}p’ filename” to retrieve that
parameter
- run the Python script with that value
> On Jul 15, 2020, at 3:13 PM, c b <breedthoughts....@gmail.com> wrote:
>
> External Email Warning
> This email originated from outside the university. Please use caution when
> opening attachments, clicking links, or responding to requests.
> I'm trying to run an embarrassingly parallel experiment, with 500+ tasks that
> all differ in one parameter. e.g.:
>
> job 1 - script.py foo
> job 2 - script.py bar
> job 3 - script.py baz
> and so on.
>
> This seems like a case where having a slurm array hold all of these jobs
> would help, so I could just submit one job to my cluster instead of 500
> individual jobs. It seems like sarray is only set up for varying an integer
> index parameter. How would i do this for non-numeric values (say, if the
> parameter I'm varying is a string in a given list) ?
>
>
