Hi Sam,
not sure I get you right how you achieve that. My code is here
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/resource.h>
#include <slurm/spank.h>
SPANK_PLUGIN(dc_test, 1);
int slurm_spank_init (spank_t sp, int ac, char **av) {
int i, argc;
char function[50];
char **argv;
char **env;
if (spank_context () != S_CTX_ALLOCATOR) {
return (0);
}
sprintf(function, "slurm_spank_init");
if (spank_get_item(sp, S_JOB_ARGV, &argc, &argv) == ESPANK_SUCCESS) {
for (i = 0; i < argc; i++) {
slurm_info( "%s (ARGV): %s\n", function, argv[i]);
}
} else {
slurm_error("Unable to retrieve S_JOB_ARGV");
}
return (0);
}
Then submit job as user:
$ sbatch test.sh
sbatch: error: Unable to retrieve S_JOB_ARGV
Submitted batch job 766
So, no cigar yet.
Kind regards,
Dmitry
Kind regards,
Dmitry Chirikov
On Mon, Jan 30, 2017 at 6:59 PM, Sam Gallop (NBI) <[email protected]>
wrote:
> Hi,
>
>
>
> By changing the entry point to slurm_spank_init I was able to retrieve the
> following (see below), but this was only when the context was
> S_CTX_ALLOCATOR. When the context is != S_CTX_ALLOCATOR I got nothing.
>
>
>
> slurm_spank_init (ARGV): sleep
>
> slurm_spank_init (ARGV): 30
>
> slurm_spank_init (ENV): SLURM_CHECKPOINT_IMAGE_DIR=/root
>
> slurm_spank_init (ENV): SLURM_NODELIST=centos1
>
> slurm_spank_init (ENV): SLURM_JOB_NAME=slurm1.sh
>
> slurm_spank_init (ENV): SLURMD_NODENAME=centos1
>
> slurm_spank_init (ENV): SLURM_TOPOLOGY_ADDR=centos1
>
> slurm_spank_init (ENV): SLURM_PRIO_PROCESS=0
>
> slurm_spank_init (ENV): SLURM_TOPOLOGY_ADDR_PATTERN=node
>
> slurm_spank_init (ENV): SLURM_NNODES=1
>
> slurm_spank_init (ENV): SLURM_JOBID=268
>
> slurm_spank_init (ENV): SLURM_TASKS_PER_NODE=3
>
> slurm_spank_init (ENV): SLURM_JOB_ID=268
>
> slurm_spank_init (ENV): SLURM_CPUS_PER_TASK=1
>
> slurm_spank_init (ENV): SLURM_JOB_USER=root
>
> slurm_spank_init (ENV): SLURM_JOB_UID=0
>
> slurm_spank_init (ENV): SLURM_NODEID=0
>
> …
>
> slurm_spank_init (ENV): SLURM_SUBMIT_DIR=/root
>
> slurm_spank_init (ENV): SLURM_TASK_PID=2470
>
> slurm_spank_init (ENV): SLURM_CPUS_ON_NODE=3
>
> slurm_spank_init (ENV): SLURM_PROCID=0
>
> slurm_spank_init (ENV): SLURM_JOB_NODELIST=centos1
>
> slurm_spank_init (ENV): SLURM_LOCALID=0
>
> slurm_spank_init (ENV): SLURM_JOB_CPUS_PER_NODE=3
>
> slurm_spank_init (ENV): SLURM_CLUSTER_NAME=cluster
>
> slurm_spank_init (ENV): SLURM_GTIDS=0
>
> slurm_spank_init (ENV): SLURM_SUBMIT_HOST=centos1
>
> slurm_spank_init (ENV): SLURM_JOB_PARTITION=debug
>
> slurm_spank_init (ENV): SLURM_JOB_NUM_NODES=1
>
> slurm_spank_init (ENV): SLURM_MEM_PER_NODE=1024
>
> slurm_spank_init (ENV): SLURM_RLIMIT_CPU=18446744073709551615
>
> slurm_spank_init (ENV): SLURM_RLIMIT_FSIZE=18446744073709551615
>
> slurm_spank_init (ENV): SLURM_RLIMIT_DATA=18446744073709551615
>
> slurm_spank_init (ENV): SLURM_RLIMIT_STACK=10485760
>
> slurm_spank_init (ENV): SLURM_RLIMIT_CORE=0
>
> slurm_spank_init (ENV): SLURM_RLIMIT_RSS=1073741824
>
> slurm_spank_init (ENV): SLURM_RLIMIT_NPROC=15188
>
> slurm_spank_init (ENV): SLURM_RLIMIT_NOFILE=1024
>
> slurm_spank_init (ENV): SLURM_RLIMIT_MEMLOCK=65536
>
> slurm_spank_init (ENV): SLURM_RLIMIT_AS=18446744073709551615
>
> slurm_spank_init (ENV): SRUN_DEBUG=3
>
> slurm_spank_init (ENV): SLURM_UMASK=0022
>
> slurm_spank_init (ENV): SLURM_NTASKS=3
>
> slurm_spank_init (ENV): SLURM_NPROCS=3
>
> slurm_spank_init (ENV): SLURM_DISTRIBUTION=block
>
> slurm_spank_init (ENV): SLURM_CPU_BIND_VERBOSE=quiet
>
> slurm_spank_init (ENV): SLURM_CPU_BIND_TYPE=cores
>
> slurm_spank_init (ENV): SLURM_CPU_BIND_LIST=
>
> slurm_spank_init (ENV): SLURM_CPU_BIND=quiet,cores
>
> slurm_spank_init (ENV): SLURM_STEP_ID=0
>
> slurm_spank_init (ENV): SLURM_STEPID=0
>
> slurm_spank_init (ENV): SLURM_SRUN_COMM_PORT=42199
>
> slurm_spank_init (ENV): SLURM_STEP_NODELIST=centos1
>
> slurm_spank_init (ENV): SLURM_STEP_NUM_NODES=1
>
> slurm_spank_init (ENV): SLURM_STEP_NUM_TASKS=3
>
> slurm_spank_init (ENV): SLURM_STEP_TASKS_PER_NODE=3
>
> slurm_spank_init (ENV): SLURM_STEP_LAUNCHER_PORT=42199
>
> slurm_spank_init (ENV): SLURM_SRUN_COMM_HOST=…
>
>
>
> ---
>
> Sam Gallop
>
>
>
> *From:* Dmitry Chirikov [mailto:[email protected]]
> *Sent:* 30 January 2017 17:25
> *To:* slurm-dev <[email protected]>
> *Subject:* [slurm-dev] RE: SPANK and job variables/options
>
>
>
> Hi Sam,
>
>
>
> thanks for sharing. If i get it right you are enumerating variables after
> allocation (slurm_spank_task_init) on the running node, but I am really
> curious if it even possible to get those vars (or some vars) before nodes
> will be allocated:
>
>
>
> int slurm_spank_init (spank_t sp, int ac, char **av) {
>
> if (spank_context () != S_CTX_ALLOCATOR) {
>
> return (0);
>
> }
>
> ...
>
> }
>
>
>
> I believe at this step srun command line options and #SBATCH instructions
> should be parsed already or, at least, available.
>
>
>
> Kind regards,
>
> Dmitry
>
>
> Kind regards,
>
> Dmitry Chirikov
>
>
>
> On Mon, Jan 30, 2017 at 4:41 PM, Sam Gallop (NBI) <[email protected]>
> wrote:
>
> Hi Dmitry,
>
>
>
> I just recently posted this for similar query about SPANK plugins (see
> below for the example code). It helps dump some of the S_JOB_ARGV or
> S_JOB_ENV variables you are interested in. It's very basic and there is
> lots that can be done with SPANK. Example output looks like this …
>
> <snip>
>
> slurm_spank_task_init (ARGV): /bin/sleep
>
> slurm_spank_task_init (ARGV): 30
>
> …
>
> slurm_spank_task_init (ENV): SLURM_JOB_NAME=slurm1.sh
>
> slurm_spank_task_init (ENV): SLURMD_NODENAME=centos1
>
> slurm_spank_task_init (ENV): SLURM_JOBID=266
>
> slurm_spank_task_init (ENV): SLURM_JOB_PARTITION=debug
>
> slurm_spank_task_init (ENV): SLURM_JOB_NUM_NODES=1
>
> slurm_spank_task_init (ENV): SLURM_MEM_PER_NODE=1024
>
> <snip>
>
>
>
> # cat userenv.c
>
> /*
>
> * To compile:
>
> * gcc -shared -o userenv.so userenv.c
>
> *
>
> */
>
> #include <sys/types.h>
>
> #include <stdio.h>
>
> #include <stdlib.h>
>
> #include <unistd.h>
>
> #include <string.h>
>
> #include <libgen.h>
>
> #include <signal.h>
>
> #include <sys/resource.h>
>
>
>
> #include <slurm/spank.h>
>
> #include <slurm/slurm.h>
>
>
>
> SPANK_PLUGIN(userenv, 1);
>
>
>
> static int _userenv_opt_process (int val,
>
> const char *optarg,
>
> int remote);
>
> void get_env(spank_t sp, char *function);
>
>
>
> struct spank_option spank_options[] =
>
> {
>
> {
>
> "userenv",
>
> "",
>
> "helper function",
>
> 0,
>
> 0,
>
> (spank_opt_cb_f) _userenv_opt_process
>
> },
>
> SPANK_OPTIONS_TABLE_END
>
> };
>
>
>
> int slurm_spank_task_init(spank_t sp, int ac, char **av)
>
> {
>
> FILE *f;
>
> char filename[256];
>
> char function[50];
>
> int i, argc;
>
> char **argv;
>
> char **env;
>
> uint32_t s_id;
>
>
>
> sprintf(function, "slurm_spank_task_init");
>
>
>
> if (spank_get_item(sp, S_JOB_ID, &s_id) != ESPANK_SUCCESS) {
>
> slurm_info("userenv: Failed to get id");
>
> return (0);
>
> }
>
>
>
> sprintf(filename, "/tmp/%s_%d_%d", function, spank_context(),
> s_id);
>
> f = fopen(filename, "w");
>
>
>
> if (spank_get_item(sp, S_JOB_ARGV, &argc, &argv) ==
> ESPANK_SUCCESS) {
>
> for (i = 0; i < argc; i++) {
>
> fprintf(f, "%s (ARGV): %s\n", function, argv[i]);
>
> }
>
> }
>
>
>
> if (spank_get_item(sp, S_JOB_ENV, &env) == ESPANK_SUCCESS) {
>
> while (*env != NULL) {
>
> fprintf(f, "%s (ENV): %s\n", function,
> *env);
>
> ++env;
>
> }
>
> }
>
>
>
> fclose(f);
>
> }
>
>
>
> static int _userenv_opt_process(int val, const char *optarg, int remote)
>
> {
>
> slurm_info("userenv: userenv plugin has been registered");
>
> return (0);
>
> }
>
>
>
> ---
>
> Sam Gallop
>
>
>
> [image: Description: Macintosh HD:Users:fretter:Documents:SugarSync Shared
> Folders:NBI:pics for hpc wiki:CiSlogo578x293.png]
>
> Have you tried looking through our *Documentation Portal*
> <http://docs.cis.nbi.ac.uk/display/CIS/CiSSupport+Home>
>
> Our documentation isn’t all text, check out our *Video Tutorials*
> <http://docs.cis.nbi.ac.uk/display/CIS/Video+tutorials>
>
> Have you tried looking through *CiS Service Desk*
> <http://support.cis.nbi.ac.uk/>
>
> Keep up to date with availability at *CiS Service Status*
> <http://status.cis.nbi.ac.uk/>
>
> More information on our HPC, Linux, Storage at *HPC Support*
> <http://docs.cis.nbi.ac.uk/display/CIS/CiSSupport+Home>* Site*
>
>
>
> To speak to us about technical issues feel free to call the *Computing
> infrastructure for Science* team on *group phone extension * *2003**.*
>
> If your request is urgent, please contact the *NBIP Computing Helpdesk*
> at [email protected] or call *phone extension **1234**.*
>
>
>
> *From:* Dmitry Chirikov [mailto:[email protected]]
> *Sent:* 26 January 2017 12:04
> *To:* slurm-dev <[email protected]>
> *Subject:* [slurm-dev] SPANK and job variables/options
>
>
>
> Hi all,
>
>
>
> Playing with SPANK I faced up with an issue
> Seems I can't get any useful bits about job which user is about to submit.
>
> I am trying to add a hook to allocator context in slurm_spank_init
> or slurm_spank_init_post_opt functions.
>
>
>
> Idea was to check some option of the committed job file (or srun options)
> and make a decision about should it be accepted to be queued or not:
>
> But seems like it is not possible to get very basic variables
> like S_JOB_ARGV or S_JOB_ENV
>
>
> If i get it right this check limits the variables spank able to use to,
> literally, slurm version.
> https://github.com/SchedMD/slurm/blob/d7770b9b68d992f4f38417d5ef7204
> 895954eff8/src/common/plugstack.c#L1765
>
>
>
> Prolog facilies were looking promised, but they cannot drop the job - it
> can be re-queued only.
>
>
>
> So the question is, how to do it properly, when variables are available to
> use, and it is not too late to deny user to submit a job?
>
>
>
> Thank you in advance.
>
>
> Kind regards,
>
> Dmitry Chirikov
>
>
>
