[slurm-users] slurmrestd health check

Hi there, we use haproxy to distribute SLURM REST API requests to multiple instances of slurmrestd. For the haproxy we need a health check. At the moment we are just checking that we get a 401 status. This works but we are ending up with a lot of noise in the log files. It would be very nice if th

[slurm-users] slurmrestd connect to 192.168.87.113:6819 Connection refused

hi，slurm configured primary and secondary，The error when requesting slurmrest api is as follows, may I ask what is the reason? # scontrol ping Slurmctld(primary) at node003 is UP Slurmctld(backup) at node113 is UP # systemctl status slurmrestd.service ● slurmrestd.service - Slurm REST daemon

[slurm-users] slurmrestd memory leak

Has anyone else noticed this issue and knows more about it? https://bugs.schedmd.com/show_bug.cgi?id=16976 Mitigation by preventing users submitting many jobs works, but only to a point.

Re: [slurm-users] slurmrestd service broken by 22.05.07 update

On 29/12/22 11:31 am, Timo Rothenpieler wrote: Having service files in top level dirs like /run or /var/lib is bound to cause issues like this. You can use local systemd overrides for things like this. In this case I suspect you can create this directory: /etc/systemd/system/slurmrestd.serv

Re: [slurm-users] slurmrestd service broken by 22.05.07 update

Ideally, the systemd service would specify the User/Group already, and then also specify RuntimeDirectory=slurmrestd. It then pre-creates a slurmrestd directory in /run for the service to put its runtime files (like sockets) into, avoiding any permission issues. Having service files in top leve

Re: [slurm-users] slurmrestd service broken by 22.05.07 update

Thanks Brian! I also discovered that I can edit the service file to remove the unix socket. Doesn't seem to impact the things I'm working with anyway. But this design choice still seems strange to me that editing the service file is required. It seems like this should also be a configurable it

Re: [slurm-users] slurmrestd service broken by 22.05.07 update

I dug up my old stuff for getting it started and see that I just disabled the unix socket completely. I was never able to get it to work for the reasons you are seeing, so I enabled it in listening mode. There are comments in the service file about it, but to do so, I changed the 'ExecStart' li

Re: [slurm-users] slurmrestd service broken by 22.05.07 update

Greetings, Thanks for responding! On 12/28/22 20:35, Brian Andrus wrote: I suspect if you delete /var/lib/slurmrestd.socket and then start slurmrestd, it will create it as the user you need it to be. Or just change the owner of it to the slurmrestd owner. No go on that. Because /var/lib re

Re: [slurm-users] slurmrestd service broken by 22.05.07 update

I suspect if you delete /var/lib/slurmrestd.socket and then start slurmrestd, it will create it as the user you need it to be. Or just change the owner of it to the slurmrestd owner. I have been running slurmrestd as a separate user for some time. Brian Andrus On 12/28/2022 3:20 PM, Chris Sta

[slurm-users] slurmrestd service broken by 22.05.07 update

Greetings, After updating to 22.05.07 (manually built from source)... slurmrestd[68695]: fatal: _create_socket: [unix:/var/lib/slurmrestd.socket] Unable to bind UNIX socket: Permission denied Looking at release notes: > -- slurmrestd - switch users earlier on startup to avoid sockets being

[slurm-users] slurmrestd does not show GrpTRESMins billing association

Hi All, I am trying to get the equivalent of this line from *scontrol show association *via slurmrestd: GrpTRESMins=cpu=N(58),mem=N(178688),energy=N(0),node=N(29), *billing=7(58),* fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0),license/interactive=N(0) But while cpu and mem are shown in bot

Re: [slurm-users] Slurmrestd authentication failed: Unspecified error

d > Environment="SLURM_JWT=daemon" > ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820 > ExecReload=/bin/kill -HUP $MAINPID > > [Install] > WantedBy=multi-user.target > > > Guillaume > > > > -- > *De: *&quo

Re: [slurm-users] Slurmrestd authentication failed: Unspecified error

uot;SLURM_JWT=daemon" ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820 ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target Guillaume De: "Chenyang Yan" À: slurm-us...@schedmd.com Envoyé: Samedi 19 Mars 2022 14:09:06 Objet: [slurm-user

[slurm-users] Slurmrestd authentication failed: Unspecified error

Hello, I have met a similar issue with slurmrestd authentication failed error, similar question: https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine ``` [root@slurmctl supervisor]# ls -l

Re: [slurm-users] Slurmrestd error on slurmdb request

slurmrestd hosts have to be identical. Cheers, Frank > From: "Philippe Noel" > To: slurm-users@lists.schedmd.com > Sent: Friday, 3 December, 2021 14:50:20 > Subject: [slurm-users] Slurmrestd error on slurmdb request > Hello, > I'm trying to send request to my

Re: [slurm-users] Slurmrestd JWT authentication [SEC=UNOFFICIAL]

Double-check you have all the packages. When slurm is built, slurmrestd is a separate package and is only built if the whole set was directed to do so. If they did not build it, you will need to do so yourself. This will mean using your custom built files throughout. Brian Andrus On 3/7/202

[slurm-users] Slurmrestd JWT authentication [SEC=UNOFFICIAL]

Hi slurm-users, Hoping you can point me in the right direction….I am trying to setup the Slurm REST API under parallelcluster v3.1.1. I am following the instructions at https://slurm.schedmd.com/jwt.html. I have added the AuthAltTypes=rest_auth/jwt and the AuthAultParameters=jwt_key=… to /opt/s

[slurm-users] slurmrestd with RS256 tokens

Has anyone got this to work? I have HS256 working fine, but when I try RS256 I get an error that the token is missing the kid field This is the decoded token: {'exp': 1644350831, 'iat': 1644343631, 'sub': 'slurm', 'kid': 'grm', 'alg': 'RS256'} I can see in the code where the error is being throw

[slurm-users] Slurmrestd error on slurmdb request

Hello, I'm trying to send request to my slurmrestd server to get all jobs: $ curl localhost:6820/slurmdb/v0.0.36/jobs --header "X-SLURM-USER-NAME: slurm" --header "X-SLURM-USER-TOKEN: e...sM" -i HTTP/1.1 200 OK Content-Length: 430 Content-Type: application/json { "meta": { "plugin": {

[slurm-users] slurmrestd features

Hi, Is there a plan for slurmrestd to get expanded? The current api is missing a few pieces of information. for example: https://slurm.schedmd.com/rest_api.html#slurmctldGetNodes does not return how many cores are allocated or how much memory is in use. Ideally this should return all the informati

Re: [slurm-users] Slurmrestd unspecified errors.

No problem. You may want to set your variables in your /etc/sysconfig/slurmrestd file. That is where you can set that variable along with others (SLURMRESTD_LISTEN, SLURMRESTD_DEBUG, SLURMREST_OPTIONS) and your service file will pick them up. Brian Andrus On 6/14/2021 12:05 PM, Heitor wrot

Re: [slurm-users] Slurmrestd unspecified errors.

On Mon, 14 Jun 2021 11:25:52 -0700 Brian Andrus wrote: > Using v20.11.7 > > I have 8081 because that is the port I am running slurmrestd on. > > How are you starting slurmrestd? If you are using systemd and have > the service file, look inside it. I'm using systemd: $ cat /usr/lib/sys

Re: [slurm-users] Slurmrestd unspecified errors.

Using v20.11.7 I have 8081 because that is the port I am running slurmrestd on. How are you starting slurmrestd? If you are using systemd and have the service file, look inside it. Brian Andrus On 6/14/2021 9:48 AM, Heitor wrote: On Mon, 14 Jun 2021 08:30:51 -0700 Brian Andrus wrote: Yo

Re: [slurm-users] Slurmrestd unspecified errors.

On Mon, 14 Jun 2021 08:30:51 -0700 Brian Andrus wrote: > You don't use the prefix. > > This works for me on the node running slurmrestd on port 8081: > > user=someuser > curl --header "X-SLURM-USER-NAME: ${user}" --header > "X-SLURM-USER-TOKEN: $(sudo scontrol toker username=${user}|cut > -d='=

Re: [slurm-users] Slurmrestd unspecified errors.

You don't use the prefix. This works for me on the node running slurmrestd on port 8081: user=someuser curl --header "X-SLURM-USER-NAME: ${user}" --header "X-SLURM-USER-TOKEN: $(sudo scontrol toker username=${user}|cut -d='=' -f2-)" http://localhost:8081/slurm/v0.0.36/ping Brian Andrus On 6

[slurm-users] Slurmrestd unspecified errors.

Hello, So far I've been unable to use slurmrestd. I'm running CentOS7 with slurm 20.11.7 from the EPEL7 repo. I generate a token this way: $ sudo scontrol token username=ubuntu SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjM2ODA2NjUsImlhdCI6MTYyMzY3ODg2NSwic3VuIjoidWJ1bnR1In0.bNIY

Re: [slurm-users] slurmrestd

Hi, On 7/06/2021 04:33, David Schanzenbach wrote: > In our .rpmmacros file we use, the following option is set: > %_with_slurmrestd 1 You also need libjwt: https://bugs.schedmd.com/show_bug.cgi?id=4 Ward

Re: [slurm-users] slurmrestd

Hi Sid, In our .rpmmacros file we use, the following option is set: %_with_slurmrestd 1 The slurm.spec file in the source tar.bz2 indicates this maps to the build option of --with slurmrestd You also need to make sure you have the build dependencies as well, which I believe are json-c-devel an

[slurm-users] slurmrestd

Hi all, I'm interested in using the slurmrestd but it does not appear to be built when you do an rpmbuild reading though the docs does not indicate a switch needed to include it (unless I missed that)... any ideas on how the rpm is built? Sid Young W: https://off-grid-engineering.com W: (

Re: [slurm-users] slurmrestd configuration

Hi Simone, On 9/04/2021 18:03, Simone Riggi wrote: > All of them are working. > So in this case the only requirement for a user is having the read/write > permission on the socket?  Correct. The authentication is done as you know the user with a socket. > My goal at the end would be to let a Do

Re: [slurm-users] slurmrestd configuration

Thanks Ward. I did the following: $ slurmrestd unix:/var/run/slurmrestd.socket -a rest_auth/local -s v0.0.36 -vvv and to test the response: $ curl --unix-socket /var/run/slurmrestd.socket localhost/slurm/v0.0.36/ping (here "localhost" can be whatever character, need to check --unix-socket cu

Re: [slurm-users] slurmrestd configuration

Hi Simone, On 8/04/2021 23:23, Simone Riggi wrote: > $ scontrol token lifespan=7200 username=riggi > > How can I configure and test the other auth method (local)? I am using > jwt at the moment. > I would like a user to be always authorized to use the rest API.  local means socket (so you don't

Re: [slurm-users] slurmrestd configuration

Hi again, I think I managed to make it working with: $ slurmrestd -f /etc/slurm/slurmrestd.conf -a rest_auth/jwt -s openapi/v0.0.36 -vvv 0.0.0.0:6820 Sample request: $ curl -H "X-SLURM-USER-NAME:riggi" -H "X-SLURM-USER-TOKEN:${SLURM_JWT}" http://localhost:6820/slurm/v0.0.36/ping { "meta": {

Re: [slurm-users] slurmrestd configuration

Ok, I think I have found the plugins in: /root/rpmbuild/BUILD/slurm-20.11.5/src/slurmrestd/plugins/openapi/ I did: make make install and now I have: $ slurmrestd -s list slurmrestd: Possible OpenAPI plugins: slurmrestd: openapi/v0.0.35 slurmrestd: openapi/v0.0.36 slurmrestd: openapi/dbv0.0.36

Re: [slurm-users] slurmrestd configuration

Thanks very much. I found both files under: /root/rpmbuild/BUILD/slurm-20.11.5/src/slurmrestd/plugins/auth/jwt/.libs/rest_auth_jwt.so /root/rpmbuild/BUILD/slurm-20.11.5/src/slurmrestd/plugins/auth/local/.libs/rest_auth_local.so It seems they weren't installed (any hints why?). By the way I tried

Re: [slurm-users] slurmrestd configuration

Hi Simone, On 8/04/2021 15:53, Simone Riggi wrote: > - I see effectively that --with jwt is not listed. I wonder how to build > (using rpmbuild) slurm auth plugins?  >   In general I didn't understand from the doc what plugins slurmrestd > expects by default and where it searches it. From -a opti

Re: [slurm-users] slurmrestd configuration

Thanks very much for your reply. - I added this line to the slurm.conf file: AuthAltParameters=jwt_key=/var/spool/slurmctld/jwt_hs256.key I restarted the daemons but this does not change slurmrestd error. I guess this is needed anyway after plugins are found, so thanks for pointing that out. -

Re: [slurm-users] slurmrestd configuration

Hi Ole, On 8/04/2021 10:09, Ole Holm Nielsen wrote: > On 4/8/21 9:50 AM, Simone Riggi wrote: >> >> rpmbuild -ta slurm-20.11.5.tar.bz2 --with mysql --with slurmrestd >> --with jwt > > I don't see this "--with jwt" in the slurm.spec file: It's not yet there: https://bugs.schedmd.com/show_bug.cgi?i

Re: [slurm-users] slurmrestd configuration

On 4/8/21 9:50 AM, Simone Riggi wrote: I write you about how to properly setup slurmrestd. ... 2) Installed slurm with: rpmbuild -ta slurm-20.11.5.tar.bz2 --with mysql --with slurmrestd --with jwt I don't see this "--with jwt" in the slurm.spec file: [slurm-20.11.5]# grep "# --with" slurm.s

Re: [slurm-users] slurmrestd configuration

Hi Simone, On 8/04/2021 09:50, Simone Riggi wrote: > where /etc/slurm/slurmrestd.conf > > include /etc/slurm/slurm.conf > AuthType=auth/jwt Did you add a key? AuthAltParameters=jwt_key=/etc/slurm/jwt.key It needs to be present on the slurmdbd and slurmctld nodes. Ward

[slurm-users] slurmrestd configuration

Dear all, I write you about how to properly setup slurmrestd. I followed a recent post ( https://groups.google.com/g/slurm-users/c/VDeFmSDao1U/m/Yu0j-VNcBgAJ) but slurmrestd fails with: *slurmrestd: fatal: No authentication plugins to load* Indeed when I run slurmrestd -a list, no plugin is detec