Hello,
We are soon to install new Slurm cluster at our site. That means that we will
have a total of three clusters running Slurm. Only two, that is the new
clusters, will share a common file system. The original cluster has its own
file system is independent of the new arrivals. If possible, w
Hello,
These days we have now enabled topology aware scheduling on our Slurm cluster.
One part of the cluster consists of two racks of AMD compute nodes. These racks
are, now, treated as separate entities by Slurm. Soon, we may add another set
of AMD nodes with slightly difference CPU specs to
Hello,
Last year I posted on this forum looking for some help on backfill in Slurm. We
are currently using Slurm 19.05.8 and we find that backfilled (smaller) jobs
tend to push back large jobs in our cluster. Chris Samuel replied to our post
with the following response...
This sounds like a pr
UTION: This e-mail originated outside the University of Southampton.
Hi David,
On 9/12/20 3:35 am, David Baker wrote:
> We see the following issue with smaller jobs pushing back large jobs. We
> are using slurm 19.05.8 so not sure if this is patched in newer releases.
This sounds like a p
9/12/20 3:35 am, David Baker wrote:
> We see the following issue with smaller jobs pushing back large jobs. We
> are using slurm 19.05.8 so not sure if this is patched in newer releases.
This sounds like a problem that we had at NERSC (small jobs pushing back
multi-thousand node jobs), and we
Hello,
We see the following issue with smaller jobs pushing back large jobs. We are
using slurm 19.05.8 so not sure if this is patched in newer releases. With a 4
node test partition I submit 3 jobs as 2 users
ssh hpcdev1@navy51 'sbatch --nodes=3 --ntasks-per-node=40
--partition=backfilltes
ve to wait a
while in the queue to ensure such.
On Tue, Oct 6, 2020 at 11:12 AM David Baker
mailto:d.j.ba...@soton.ac.uk>> wrote:
Hello,
I would appreciate your advice on how to deal with this situation in Slurm,
please. If I have a set of nodes used by 2 groups, and normally each group
Hello,
I would appreciate your advice on how to deal with this situation in Slurm,
please. If I have a set of nodes used by 2 groups, and normally each group
would each have access to half the nodes. So, I could limit each group to have
access to 3 nodes each, for example. I am trying to devise
Hello,
I wondered if someone would be able to advise me on how to limit access to a
group of resources, please.
We have just installed a set of 6 GPU nodes. These nodes belong to a research
department and both staff and students will potentially need access to the
nodes. I need to ensure that
\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'
> On Sep 10, 2020, at 11:00 AM, David Baker wrote:
>
> Hello,
>
> We are installing a group of nodes which all contain 4 GPU c
Hello,
We are installing a group of nodes which all contain 4 GPU cards. The GPUs are
paired together using NVLINK as described in the matrix below.
We are familiar with using Slurm to schedule and run jobs on GPU cards, but
this is the first time we have dealt with NVLINK enabled GPUs. Could s
Hello,
We are currently helping a research group to set up their own Slurm cluster.
They have asked a very interesting question about Slurm and file systems. That
is, they are posing the question -- do you need a shared user file store on a
Slurm cluster?
So, in the extreme case where this is
] Nodes do not return to service after scontrol reboot
On 6/16/20 8:16 am, David Baker wrote:
> We are running Slurm v19.05.5 and I am experimenting with the *scontrol
> reboot * command. I find that compute nodes reboot, but they are not
> returned to service. Rather they remain down foll
Hello,
We are running Slurm v19.05.5 and I am experimenting with the scontrol reboot
command. I find that compute nodes reboot, but they are not returned to
service. Rather they remain down following the reboot..
navy55 1debug*down 80 2:20:2 1920000 2000
(nu
Hello,
We have, rather belatedly, just upgraded to Slurm v19.05.5. On the whole, so
far so good -- no major problems. One user has complained that his job now
crashes and reports an unlink error. That is..
slurmstepd: error: get_exit_code task 0 died by signal: 9
slurmstepd: error: unlink(/tmp
Hello,
Before implementing "GrpTRESRunMin=cpu=limit" on our production cluster I'm
doing some tests on the development cluster. I've only get a handful of compute
nodes to play without and so I have set the limit sensibly low. That is, I've
set the limit to be 576,000. That's equivalent to 400
w they affect the
performance of your scheduler.
Any chance you could let us know how things go?
Killian
On Tue, 4 Feb 2020 at 10:43, David Baker
mailto:d.j.ba...@soton.ac.uk>> wrote:
Hello,
Thank you very much again for your comments and the details of your slurm
configuration. All t
t our backfilling was sub-par until we tuned these
parameters (or at least a few users could find a way to submit so many jobs
that the backfill couldn’t keep up, even when we had idle resources for their
very short jobs).
> On Jan 31, 2020, at 3:01 PM, David Baker wrote:
>
> External
our wait times aren’t
due to having a ton of spare capacity for extended periods of time.
Not sure how much of that will help immediately, but it may give you some ideas.
> On Jan 31, 2020, at 10:14 AM, David Baker wrote:
>
> External Email Warning
> This email originated from outside
vices
931 372-3601 / Tennessee Tech University
On Jan 31, 2020, at 6:23 AM, David Baker wrote:
Hello,
Our SLURM cluster is relatively small. We have 350 standard compute nodes each
with 40 cores. The largest job that users can run on the partition is one
requesting 32 nodes. Our clust
Hello,
Our SLURM cluster is relatively small. We have 350 standard compute nodes each
with 40 cores. The largest job that users can run on the partition is one
requesting 32 nodes. Our cluster is a general university research resource and
so there are many different sizes of jobs ranging from
098.ba+ batch default 12 OUT_OF_ME+0:125
10816098.ex+ extern default 12 COMPLETED 0:0
10816098.0 vasp_mpi default 12 OUT_OF_ME+0:125
Best
Marcus
On 11/7/19 5:36 PM, David Baker wrote:
Hello,
We are deali
Hello,
We are dealing with some weird issue on our shared nodes where job appear to be
stalling for some reason. I was advised that this issue might be related to the
oom-killer process. We do see a lot of these events. In fact when I started to
take a closer look this afternoon I noticed that
agner
Sent: 06 November 2019 09:53
To: David Baker ; slurm-users@lists.schedmd.com
; juergen.s...@uni-ulm.de
Subject: Re: [slurm-users] Running job using our serial queue
Hi David,
if I remember right (we have disabled swap for years now), swapping out
processes seem to slow down the system ov
EX and Gurobi, both often used from Matlab. So even,
if the user sets '-singleCompThread' for Matlab, that does not mean at all, the
job is only using one CPU.
Best
Marcus
On 11/4/19 4:14 PM, David Baker wrote:
Hello,
We decided to route all jobs requesting from 1 to 20 cores to our se
Hello,
We decided to route all jobs requesting from 1 to 20 cores to our serial queue.
Furthermore, the nodes controlled by the serial queue are shared by multiple
users. We did this to try to reduce the level of fragmentation across the
cluster -- our default "batch" queue provides exclusive a
Hello.,
I have just started to take a look at Slurm v19* with a view to an upgrade
(most likely in the next year). My reaction is that Slurm very rarely provides
an estimated start time for a job. I understand that this is not possible for
jobs on hold and dependent jobs. On the other hand I've
: Slurm User Community List
Subject: Re: [slurm-users] How to modify the normal QOS
* David Baker [190926 14:12]:
>
> Currently my normal QOS specifies MaxTRESPU=cpu=1280,nodes=32. I've
> tried a number of edits, however I haven't yet found a way of
> redefining the MaxTRESP
Hello,
Currently my normal QOS specifies MaxTRESPU=cpu=1280,nodes=32. I've tried a
number of edits, however I haven't yet found a way of redefining the MaxTRESPU
to be "cpu=1280". In the past I have resorted to deleting a QOS completely and
redefining the whole thing, but in this case I'm not s
; Compute Services (SSCS)
Kommunikations- und Informationszentrum (kiz)
Universität Ulm
Telefon: +49 (0)731 50-22478
Telefax: +49 (0)731 50-22471
* David Baker [190925 12:12]:
> Hello,
>
> I have defined a partition and corresponding QOS in Slurm. This is
> the serial queue to which we ro
Hello,
I have defined a partition and corresponding QOS in Slurm. This is the serial
queue to which we route jobs that require up to (and including) 20 cpus. The
nodes controlled by serial are shared. I've set the QOS like so..
[djb1@cyan53 slurm]$ sacctmgr show qos serial format=name,maxtrespe
Hello,
I apologise that this email is a bit vague, however we are keen to understand
the role of the Slurm "StateSave" location. I can see the value of the
information in this location when, for example, we are upgrading Slurm and the
database is temporarily down, however as I note above we are
5f7dab1400983e008d710f8840c%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=bhMG78N1%2FQ2ZInn599QuEQ6tyD5pRXAIomlNja1f3j0%3D&reserved=0>
I think 18.08.04 and later have it fixed.
Jeff
____
From: slurm-users on behalf of David
Baker
Sent: Thursday, July 25, 2019 6:53 AM
To:
rm node weights
Which version of Slurm are you running? I know some of the earlier versions of
18.08 had a bug and node weights were not working.
Jeff
From: slurm-users on behalf of David
Baker
Sent: Thursday, July 25, 2019 6:09 AM
To: slurm-users@lists.sched
Hello,
As an update I note that I have tried restarting the slurmctld, however that
doesn't help.
Best regards,
David
From: slurm-users on behalf of David
Baker
Sent: 25 July 2019 11:47:35
To: slurm-users@lists.schedmd.com
Subject: [slurm-users]
Hello,
I'm experimenting with node weights and I'm very puzzled by what I see. Looking
at the documentation I gathered that jobs will be allocated to the nodes with
the lowest weight which satisfies their requirements. I have 3 nodes in a
partition and I have defined the nodes like so..
Node
users] Requirement to run longer jobs
Hi Chris,
Chris Samuel writes:
> On 3/7/19 8:49 am, David Baker wrote:
>
>> Does the above make sense or is it too complicated?
>
> [looks at our 14 partitions and 112 QOS's]
>
> Nope, that seems pretty simple. We do much the sam
Hello,
A few of our users have asked about running longer jobs on our cluster.
Currently our main/default compute partition has a time limit of 2.5 days.
Potentially, a handful of users need jobs to run up to 5 hours. Rather than
allow all users/jobs to have a run time limit of 5 days I wonder
Hello,
Everyday we see several deadlocks in our slurmdbd log file. Together with the
deadlock we always see a failed "roll up" operation. Please see below for an
example.
We are running slurm 18.08.0 on our cluster. As far as we know these deadlocks
are not adversely affecting the operation
Subject: Re: [slurm-users] Advice on setting up fairshare
Hi David,
I have had time to look into your current problem, but inline I have
some comments about the general approach.
David Baker writes:
> Hello,
>
> Could someone please give me some advice on setting up the fairshare
>
Hello,
Could someone please give me some advice on setting up the fairshare in a
cluster. I don't think the present setup is wildly incorrect, however either my
understanding of the setup is wrong or something is reconfigured.
When we set a new user up on the cluster and they haven't used any
Hello,
I have a quick question regarding updating the priority flags in the
slurm.conf file. Currently I have the flag "small_relative_to_time" set.
I'm finding that that flag is washing out the job size priority weight
factor and I would like to experiment without it.
So when you remove that pri
Hello,
Following the various postings regarding slurm 19.05 I thought it was an
opportune time to send this question to the forum.
Like others I'm awaiting 19.05 primarily due to the addition of the XFACTOR
priority setting, but due to other new/improved features as well. I'm
interested to hea
Hello,
We are experiencing quite a number of database failures. We saw an outright
failure a short while ago where we had to restart the maria database and the
slurmdbd process. After restarting the database appear to be working well,
however over the last few days I have notice quite a number
utility out of that.
I'd suggest looking at the output of sprio to see how your factors are working
in situ, particularly when you've got a stuck large job. It may be that the
SMALL_RELATIVE_TO_TIME could be washing out the job size factor if your larger
jobs are also longer.
HTH.
M
elative to the other factors.
Have you looked at PriorityWeightJobSize? Might have some utility if you're
finding large jobs getting short-shrift.
- Michael
On Tue, Apr 9, 2019 at 2:01 AM David Baker
mailto:d.j.ba...@soton.ac.uk>> wrote:
Hello,
I've finally got the job th
Hello,
I've finally got the job throughput/turnaround to be reasonable in our cluster.
Most of the time the job activity on the cluster sets the default QOS to 32
nodes (there are 464 nodes in the default queue). Jobs requesting nodes close
to the QOS level (for example 22 nodes) are scheduled
2019?
The 2019 Slurm User Group Meeting will be held in Salt Lake City at the
University of Utah on September 17-18.
Registration for this user group meeting typically opens in May.
Jacob
On Mon, Mar 25, 2019 at 2:57 PM david baker
mailto:djbake...@gmail.com>> wrote:
Hello,
I was se
Hello,
I was searching the web to see if there was going to be a Slurm users’ meeting
this year, but couldn’t find anything. Does anyone know if there is a users’
meeting planned for 2019? If so, is it most likely going to be held as part of
Supercomputing in Denver? Please let me know if yo
e you some reasonable starting points.
Doug
On Sat, Mar 23, 2019 at 05:26 david baker
mailto:djbake...@gmail.com>> wrote:
Hello,
We do have large jobs getting starved out on our cluster, and I note
particularly that we never manage to see a job getting assigned a start time.
It seems ver
Hello,
We do have large jobs getting starved out on our cluster, and I note
particularly that we never manage to see a job getting assigned a start
time. It seems very possible that backfilled jobs are stealing nodes
reserved for large/higher priority jobs.
I'm wondering if our backfill configura
(Resources)
Best regards,
David
From: slurm-users on behalf of
Christopher Samuel
Sent: 21 March 2019 17:54
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] Very large job getting starved out
On 3/21/19 6:55 AM, David Baker wrote:
> it current
ata=KfjAqNHQgLcUBBYwZFi8OygU2De%2FdVuTwbdOmUv0Dps%3D&reserved=0>
as another potential starting point.
Best,
Cyrus
On 3/21/19 8:55 AM, David Baker wrote:
Hello,
I understand that this is not a straight forward question, however I'm
wondering if anyone has any useful ideas, please. Our
Hello,
I understand that this is not a straight forward question, however I'm
wondering if anyone has any useful ideas, please. Our cluster is busy and the
QOS has limited users to a maximum of 32 compute nodes on the "batch" queue.
Users are making good of the cluster -- for example one user
gt; need to do anything. If you need something less than the max memory per
> node then you will need to enforce some limits. We do this via a jobsubmit
> lua script. That would be my recommended method.
>
>
> -Paul Edmon-
>
>
> On 3/12/19 12:31 PM, David Baker wrote:
Hello,
I have set up a serial queue to run small jobs in the cluster. Actually, I
route jobs to this queue using the job_submit.lua script. Any 1 node job using
up to 20 cpus is routed to this queue, unless a user submits their job with an
exclusive flag.
The partition is shared and so I def
pended mode
> while the high priority job completes and then the low priority job
> continues from where it stopped. No checkpoints and no killing.
>
> Antony
>
>
>
> On Fri, 1 Mar 2019, 12:23 david baker, wrote:
>
>> Hello,
>>
>> Following up on implementing p
; happy.
>
> The one caveat is that jobs that will be killed and requeued need to
> support checkpoint/restart. So when this becomes a production thing, users
> are going to have to acknowledge that they will only use this partition for
> jobs that have some sort of checkpoint/restart
om lower
> priority tier partitions, without taking the normal fairshare priority into
> consideration.
>
> Best
> Marcus
>
> On 2/15/19 10:07 AM, David Baker wrote:
>
> Hello.
>
>
> We have a small set of compute nodes owned by a group. The group has
> agreed
Hello.
We have a small set of compute nodes owned by a group. The group has agreed
that the rest of the HPC community can use these nodes providing that they (the
owners) can always have priority access to the nodes. The four nodes are well
provisioned (1 TByte memory each plus 2 GRID K2 graph
Hello,
I'm sure that this question has been asked before. We have recently added
some GPU nodes to our SLURM cluster.
There are 10 nodes each providing 2 * Tesla V100-PCIE-16GB cards There are
10 nodes each providing 4 * GeForce GTX 1080 Ti cards
I'm aware that the simplest way to manage t
61 matches
Mail list logo
