You can stop tracking memory by changing
SelectTypeParameters=CR_Core_Memory to SelectTypeParameters=CR_Core
doing this will mean slurm is no longer tracking memory at all and
jobs could in theory stop on one another if they allocate to much
physical memory.
we haven't started tracking memory on our slurm clusters and have
gpu's so i'm very curious to the answer on this one as well. when our
jobs run we routinely see the virtual memory on the process spiral up
to +100GB, but the physical memory is much less
On Wed, Apr 26, 2017 at 3:41 PM, Samuel Bowerman <sbowe...@hawk.iit.edu> wrote:
> Hello Slurm community,
>
> Our lab has recently begun transitioning from maui/torque to slurm, but we
> are having some difficulties getting our configuration correct. In short,
> our CUDA tests routinely violate the virtual memory limits assigned by the
> scheduler even though the physical memory space is orders of magnitude
> smaller than the amount of memory requested through slurm. Since our
> CPU-only versions of the same program (pmemd of the Amber suite - v16) runs
> without problems, I am highly suspicious that our scheduler is not handing
> GPU memory appropriately.
>
> Here is some relevant information/specs:
> - I am a user, not the sys-admin. Both the admin and myself are new to
> managing slurm installations, but I do have experienced on the user end
> through XSEDE HPC resources.
> - Cluster runs on Rocks 6.2
> - One head node, 12 compute nodes
> - Each compute node has 12 CPU cores, 64 GB of RAM, and 4 NVIDIA GeForce GTX
> 1080 GPU's
> - GPU version of program uses less than 1 GB of physical memory (~0.6-8 GB)
>
> Here are some symptoms of the issue:
> - CPU version of program runs fine using default SLURM memory allocation
> - SBATCH option "--mem=2GB" returns an error: "cudaGetDeviceCount failed out
> of memory". This problem is only resolved when I declare "--mem=10GB". I
> find it very hard to believe that this routine requires that much memory.
> - SLURM will kill the job unless I declare "--mem=20GB" or above with the
> error message: "slurmstepd: error: Job exceeded virtual memory limit". As a
> reminder that the program in question uses less than 1GB of physical memory;
> however, the virtual memory footprint is on the order of 20g.
> - Requesting more than "--mem=62GB" results in a job never running due to
> lack of resources ("scontrol show node" returns
> "RealMemory=64527,FreeMemory=62536"), so if I understand the SLURM scheduler
> appropriately, the virtual memory limit is set according to maximum physical
> memory available (even though virtual memory usage =/= physical memory
> usage).
> - According to the previous bullet point, SLURM believes we should only be
> able to run up to 3 copies of this program at once, but our maui/torque
> nodes are currently running 4 versions of the program + other jobs
> simultaneously without any performance losses.
>
> In summary, we are looking for a way to either shut off the SLURM memory
> allocation and let the programs "self-regulate" how much memory they are
> using or to have SLURM enforce a physical memory limit instead of a virtual
> one. We have tried several things to no avail since we are both new to
> SLURM, so we are hoping for some help on the matter. Our current slurm.conf
> file is printed at the end of this email, and any assistance on the matter
> would be greatly appreciated. Thanks!
>
> Take care,
> Samuel Bowerman
> Ph.D. Candidate
> Department of Physics
> Illinois Institute of Technology
>
> Begin slurm.conf:
>
> SlurmUser=root
> SlurmdUser=root
> SlurmctldPort=6817
> SlurmdPort=6818
> AuthType=auth/munge
> CryptoType=crypto/munge
> StateSaveLocation=/var/spool/slurm.state
> SlurmdSpoolDir=/var/spool/slurmd
> SwitchType=switch/none
> MpiDefault=none
> SlurmctldPidFile=/var/run/slurmctld.pid
> SlurmdPidFile=/var/run/slurmd.pid
> ProctrackType=proctrack/linuxproc
> PluginDir=/usr/lib64/slurm
> CacheGroups=0
> JobCheckpointDir=/var/spool/slurm.checkpoint
> #SallocDefaultCommand = "xterm"
> GresTypes=gpu
> #FirstJobId=
> ReturnToService=2
> #MaxJobCount=
> #PlugStackConfig=
> #PropagatePrioProcess=
> #PropagateResourceLimits=
> #PropagateResourceLimitsExcept=
>
> PropagateResourceLimits=NONE
>
> #Prolog=
> #Epilog=
> #SrunProlog=
> #SrunEpilog=
> #TaskProlog=
> #TaskEpilog=
> TaskPlugin=task/affinity
> TrackWCKey=yes
> TopologyPlugin=topology/none
> #TreeWidth=50
> TmpFs=/state/partition1
> #UsePAM=
> SlurmctldTimeout=300
> SlurmdTimeout=300
> InactiveLimit=30
> MinJobAge=300
> KillWait=60
> WaitTime=60
> SelectType=select/cons_res
> SelectTypeParameters=CR_Core_Memory
> #DefMemPerCPU=220
> #MaxMemPerCPU=300
> VSizeFactor=110
> FastSchedule=0
> ####
> #Set MemLimitEnforce
> MemLimitEnforce=No
> ##
>
> JobCompType=jobcomp/none
> JobAcctGatherType=jobacct_gather/linux
> JobAcctGatherFrequency=30
> ##JobAcctGatherParams=UsePss
>
> ####### Priority Begin ##################
> PriorityType=priority/multifactor
> PriorityDecayHalfLife=14-0
> PriorityWeightFairshare=10000
> PriorityWeightAge=10
> PriorityWeightPartition=10000
> PriorityWeightJobSize=100
> PriorityMaxAge=1-0
> PriorityWeightQOS=0
> PriorityFlags=FAIR_TREE
> PriorityWeightTRES=cpu=2000,mem=1,gres/gpu=400
> AccountingStorageTRES=gres/gpu
> AccountingStorageEnforce=all
> FairShareDampeningFactor=5
> ####### Priority End ##################
>
> SlurmctldDebug=3
> SlurmctldLogFile=/var/log/slurm/slurmctld.log
> SlurmdDebug=3
> SlurmdLogFile=/var/log/slurm/slurmd.log
> DebugFlags=Priority,NO_CONF_HASH,backfill,BackfillMap
>
> NodeName=DEFAULT State=UNKNOWN
> NodeName=porter NodeAddr=10.1.1.1
> PartitionName=DEFAULT AllocNodes=porter State=UP
> PartitionName=DEBUG
>
> ####### Power Save Begin ##################
> # SuspendExcNodes=<your headnode>,compute-1-0
> SuspendProgram=/etc/slurm/suspendhost.sh
> SuspendRate=4
> # SuspendTime = 600
> SuspendTime=-1
> SuspendTimeout=45
> ResumeProgram=/etc/slurm/resumehost.sh
> ResumeRate=4
> ResumeTimeout= 450
> ####### Power Save End ##################
>
> ################ Do not edit below
> #############################################################
> include /etc/slurm/headnode.conf
> include /etc/slurm/nodenames.conf
> include /etc/slurm/partitions.conf
> ################################################################################################
>
