[slurm-users] Starting a job after a file is created in previous job (dependency looking for soluton)

2024-02-06 Thread Amjad Syed via slurm-users
Hello I have the following scenario: I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is gua

Re: [slurm-users] Decreasing time limit of running jobs (notification)

2023-07-06 Thread Amjad Syed
nfirmation if you are setting a max time that is less than the current > walltime? Perhaps. Could you script that yourself? Yes, I’m certain of it. > Those kind of built-in safeguards aren’t super common, however. > > Jason > > On Thu, Jul 6, 2023 at 12:55 PM Amjad Syed wrote: > &

Re: [slurm-users] Decreasing time limit of running jobs (notification)

2023-07-06 Thread Amjad Syed
unfortunate typo. > > Jason > > On Thu, Jul 6, 2023 at 11:54 AM Amjad Syed wrote: > >> Hello >> >> We were trying to increase the time limit of a slurm running job >> >> scontrol update job= TimeLimit=16-00:00:00 >> >> But we accidentally

[slurm-users] Decreasing time limit of running jobs (notification)

2023-07-06 Thread Amjad Syed
Hello We were trying to increase the time limit of a slurm running job scontrol update job= TimeLimit=16-00:00:00 But we accidentally got it to 16 hours scontrol update job= TimeLimit=16:00:00 This actually timeout and killed the running job and did not give any notification Is this a bug, sh

[slurm-users] Secondary Unix group id of users not being issued in interactive srun command

2021-09-21 Thread Amjad Syed
Hello all We have users who have have defined unix secondary id on our login nodes. vas20xhu@login01 ~]$ groups BIO_pg BIO_AFMAKAY_LAB_USERS But when we run interactive and go to compute node , the user does not have secondary group of BIO_AFMAKAY_LAB_USERS vas20xhu@c0077 ~]$ groups BIO_

Re: [slurm-users] [EXT] User association with partition and Qos

2021-08-31 Thread Amjad Syed
Just a correction We use sacctmgr modify user= set qos+=gpu-rtx6000-2 Amjad On Tue, Aug 31, 2021 at 10:17 AM Amjad Syed wrote: > Hi Sean > > We have been adding by using the following command > > sacctmgr modify user set qos+=gpu-rtx-reserved > > We have a single accou

Re: [slurm-users] [EXT] User association with partition and Qos

2021-08-31 Thread Amjad Syed
n > ------ > *From:* slurm-users on behalf of > Amjad Syed > *Sent:* Tuesday, 31 August 2021 17:46 > *To:* Slurm User Community List > *Subject:* Re: [slurm-users] [EXT] User association with partition and Qos > > * External email: Please exercise cau

Re: [slurm-users] [EXT] User association with partition and Qos

2021-08-31 Thread Amjad Syed
t does Slurm show for the partition config? > > sacctmgr show account withassoc -p > scontrol show part gpu-rtx6000-2 > > Sean > -- > *From:* slurm-users on behalf of > Amjad Syed > *Sent:* Tuesday, 31 August 2021 17:03 > *To:* Slurm User Com

Re: [slurm-users] [EXT] User association with partition and Qos

2021-08-31 Thread Amjad Syed
slurm.conf so that qos becomes permanent ? Amjad On Fri, Aug 27, 2021 at 3:32 PM Amjad Syed wrote: > Hi Sean, > > Thanks for the suggestion, seems to work now. > > Majid > > On Fri, Aug 27, 2021 at 12:56 PM Sean Crosby > wrote: > >> Hi Amjad, >> >&g

Re: [slurm-users] [EXT] User association with partition and Qos

2021-08-27 Thread Amjad Syed
its,qos,safe > > Sean > > -- > *From:* slurm-users on behalf of > Amjad Syed > *Sent:* Friday, 27 August 2021 20:28 > *To:* slurm-us...@schedmd.com > *Subject:* [EXT] [slurm-users] User association with partition and Qos > > * External email: Please exerc

[slurm-users] User association with partition and Qos

2021-08-27 Thread Amjad Syed
Hello all We are having an issue understanding user association and partition. Currently we have a partition with 30 GPU cards . We have defined a qos gpu-rtx that allows user to reserve 2 cards sacctmgr show qos gpu-rtx format=MaxTRESPU%60 Ma

[slurm-users] Submit time instead of Start time for sacct

2021-08-09 Thread Amjad Syed
Hello all, I am trying to filter number of jobs submitted in a month , not jobs that started . if i use sacct -S 2021-07-07 -E 2021-08-07 --format=jobID,Submit -D JobID Submit --- 7274903 2021-06-09T11:30:46 I get jobs that were su

[slurm-users] sreport getting last 30 reports

2021-07-26 Thread Amjad Syed
Hello I am trying to find what is the best way of using Start=now - 30 days in sreport I have used sreport User TopUsage Start=now-30Days But it does not seem to take it. Any thing i am missing here? Majid

[slurm-users] Effect of slurmctld and slurmdb going down on running/pending jobs

2021-06-23 Thread Amjad Syed
Hello all We have a cluster running centos 7 . Our slurm scheduler is running on a vm machine and we are running out of disk space for /var The slurm innodb is taking most of space. We intend to expand the vdisk for slurm server. This will require a reboot for changes to take effect. D