Re: [slurm-users] Virtual memory size requested by slurm

>If you want the virtual memory size to be unrestricted by slurm, set VSizeFactor to 0 in slurm.conf, which according >to the documentation disables virtual memory limit enforcement. > >https://slurm.schedmd.com/slurm.conf.html#OPT_VSizeFactor

Re: [slurm-users] Virtual memory size requested by slurm

On this part, I don’t think that’s always the case. On a node with 384 GB (with 2 GB reserved for the OS), we’ve got several jobs running under mem=32000: = $ grep 'NodeName=gpunode\[00' /etc/slurm/slurm.conf NodeName=gpunode[001-003] CoresPerSocket=14 RealMemory=382000 Sockets=2 ThreadsPe

Re: [slurm-users] Virtual memory size requested by slurm

Hi Mahmood, If you want the virtual memory size to be unrestricted by slurm, set VSizeFactor to 0 in slurm.conf, which according to the documentation disables virtual memory limit enforcement. https://slurm.schedmd.com/slurm.conf.html#OPT_VSizeFactor -Sean On Mon, Jan 27, 2020 at 11:47 PM Mahmo

Re: [slurm-users] Virtual memory size requested by slurm

>This line is probably what is limiting you to around 40gb. >#SBATCH --mem=38GB Yes. If I change that value, the "ulimit -v" also changes. See below [shams@hpc ~]$ cat slurm_blast.sh | grep mem #SBATCH --mem=50GB [shams@hpc ~]$ cat my_blast.log virtual memory (kbytes, -v) 57671680 /var/

Re: [slurm-users] Virtual memory size requested by slurm

>You may check here and see if it helps you: >https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd I guess these settings are related to the system. As I said, outsid

Re: [slurm-users] Virtual memory size requested by slurm

Hi Mahmood, You may check here and see if it helps you: https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd On Mon, Jan 27, 2020 at 2:41 PM Mahmood Naderan wrote: > Excuse me, I see that "ulimit -a" shows unlimited virtual memory size when > the user runs

Re: [slurm-users] Virtual memory size requested by slurm

Excuse me, I see that "ulimit -a" shows unlimited virtual memory size when the user runs that on terminal. However, when he puts the command in sbatch script, the value is limited to about 40GB. [shams@hpc ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) un

Re: [slurm-users] Virtual memory size requested by slurm

>alternatively, you can ask slurm not to limit VSZ: in cgroup.conf, have >ConstrainSwapSpace=no >this does not actually permit arbitrary VSZ, since there are mechanisms >outside the cgroup limit that affect max VSZ (overcommit sysctls, swap space) Hi Mark, ConstrainSwapSpace=no or ConstrainSwapSp

Re: [slurm-users] Virtual memory size requested by slurm

As a follow up to my last problem, I would like to know how can I tell slurm to increase the virtual memory size for a process? have you read the messages on this list? first, you can ask for the correct amount of memory. this approach assumes that it is dangerous to allow VSZ > RSS. that's c

[slurm-users] Virtual memory size requested by slurm

Hi, As a follow up to my last problem, I would like to know how can I tell slurm to increase the virtual memory size for a process? The program has no problem when I run it outside of the slurm via terminal/bash. Regards, Mahmood