All,
The folks here at NCCS helped me with a possible way. I do something
like this:
srun --output=srunps.%N-job%j-%t.out --jobid=5102027 'ps -ef'
So that it puts each node's ps in a separate file delineated by the
hostname.
Oddly, I can't seem to figure out how to pipe inside that call: issuing
'ps -ef' does the same as 'ps -ef | grep GEOS'.
Still, it works pretty well!
Matt
On 08/26/2015 12:05 PM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND
APPLICATIONS INC] wrote:
Nope. That doesn't seem to work:
(4607) $ squeue -l -u mathomp4
Wed Aug 26 12:03:44 2015
JOBID PARTITION NAME USER STATE TIME
TIMELIMIT NODES NODELIST(REASON)
5096279 compute EnADAS-I mathomp4 RUNNING 3:49
1:00:00 8 borgo[046-053]
5095474 compute Interact mathomp4 RUNNING 3:11:09
9:00:00 8 borgo[037-041,043-045]
So I can try on borgo046 which is in batch:
(4608) $ srun -w borgo046 ps
srun.slurm: Required node not available (down or drained)
srun.slurm: job 5096288 queued and waiting for resources
Looks like it wants to try and allocate a new allocation...so it'll wait
until the job is done.
And since -w == --nodelist, it doesn't work with --jobid.
Matt
On 08/26/2015 11:24 AM, Aaron Knister wrote:
Try srun -w $NODENAME
Sent from my iPhone
On Aug 26, 2015, at 11:06 AM, Thompson, Matt[SCIENCE SYSTEMS AND
APPLICATIONS INC] (GSFC-610.1) <[email protected]> wrote:
SLURM Dev,
I'm hoping you can help me with something. I recently had need to
figure out what's going on inside a running batch job. I had a
suspicion that maybe a job that should use, say, 48 cores on a few
nodes managed to somehow pack them all on a single node due to my
idiocy with an mpirun command.
So, I have a job-id and I know I can do, say:
srun --jobid=<JOBID> ps -ef
and I'll get a ps on the nodes in that allocation. But, if that
allocation has, say, 14 nodes, I get 14 nodes worth of information
that is hard to parse out since ps doesn't prepend/print hostname[1].
I thought maybe there is a way to run the srun command on just one of
the nodes in the allocation and I tried:
srun --jobid=<JOBID> --nodelist=node1 ps -ef
where node1 is one of the nodes in the allocation. But, no, that
doesn't seem to do what I'd hoped as I still get every node running ps.
Now, I'm sure I could whip up a bash script which tests for the
hostname and runs a command only if that matches the one I want, but
I was hoping for a nice simple way with srun itself to do this.
Matt
[1] That I know of. I didn't see "hostname" in the ps manpage.
--
Matt Thompson SSAI, Sr Software Test Engr
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
--
Matt Thompson SSAI, Sr Software Test Engr
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
