Hi John,
thanks for the reply :)
Yes I've got logrotate enabled for my slurm :
/var/log/slurm/slurmd.log /var/log/slurm/slurmctld.log
/var/log/slurm/slurmdbd.log {
compress
missingok
nocopytruncate
nocreate
nodelaycompress
nomail
notifempty
noolddir
rotate 12
sharedscripts
size=5M
postrotate
/usr/sbin/invoke-rc.d --quiet slurm-llnl reconfig >/dev/null
endscript
}
If I can't use logrotate, what must I use ?
I just disabled it, I'm gonna see if the problem still persist.
* My SlurmctldTimeout is set to 300, in the doc it seems like it's in
seconds, so 300 seconds for a timeout is way enough I hope !
* If there's typos, slurm doesn't warn for them at start or somewhere later
in the logs ?
* No upgrade since the compilation.
Thanks for reply, if you have any other idea, don't hesitate to share !
Regards,
On Mon, Sep 26, 2016 at 9:39 PM, John DeSantis <[email protected]> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Philippe,
>
> > But every 3 days, the slurmctld process restarts by itself, as you
> > can see in slurmctld.log :
>
> ... SNIP ...
>
> > No crontab set, anything, it just restarts itself. And the thing
> > is, when I got jobs running for more than 3 days, they are killed
> > by this restart (even if, normally, slurm is capable to resume
> > jobs)
>
> Do you have log rotation enabled that is stopping and restarting the ctl
> d?
>
> As far as the lost jobs go, check your 'SlurmctldTimeout' and see if
> it's set too low. We've never lost any jobs due to:
>
> * ctld restarts
> * typos in slurm.conf (!!)
> * upgrades
>
> I've been especially guilty of typos, and FWIW SLURM has been
> extremely forgiving.
>
> HTH,
> John DeSantis
>
>
>
> On 09/26/2016 03:46 AM, Philippe wrote:
> > Hello everybody, I'm trying to understand an issue with 2 SLURM
> > installations on Ubuntu 14.04 64b, with slurm 2.6.9 compiled. Only
> > 1 computer running Slurmctld/slurmd/slurmdbd. It works very well
> > for any job that doesn't last more than 2 days.
> >
> > But every 3 days, the slurmctld process restarts by itself, as you
> > can see in slurmctld.log :
> >
> > [2016-09-26T08:01:42.682] debug2: Performing purge of old job
> > records [2016-09-26T08:01:42.683] debug: sched: Running job
> > scheduler [2016-09-26T08:01:42.683] debug2: Performing full system
> > state save [2016-09-26T08:01:44.003] debug: slurmdbd: DBD_RC is -1
> > from DBD_FLUSH_JOBS(1408): (null) [2016-09-26T08:01:44.751] debug2:
> > Sending cpu count of 22 for cluster [2016-09-26T08:01:44.792]
> > debug: slurmdbd: Issue with call DBD_CLUSTER_CPUS(1407):
> > 4294967295(This cluster hasn't been added to accounting yet)
> > [2016-09-26T08:01:46.792] debug2: Testing job time limits and
> > checkpoints [2016-09-26T08:01:47.005] debug3: Writing job id 840 to
> > header record of job_state file [2016-09-26T08:02:03.003] debug:
> > slurmdbd: DBD_RC is -1 from DBD_FLUSH_JOBS(1408): (null)
> > [2016-09-26T08:02:16.582] Terminate signal (SIGINT or SIGTERM)
> > received [2016-09-26T08:02:16.582] debug: sched: slurmctld
> > terminating [2016-09-26T08:02:16.583] debug3: _slurmctld_rpc_mgr
> > shutting down [2016-09-26T08:02:16.798] Saving all slurm state
> > [2016-09-26T08:02:16.801] debug3: Writing job id 840 to header
> > record of job_state file [2016-09-26T08:02:16.881] debug3:
> > _slurmctld_background shutting down [2016-09-26T08:02:16.952]
> > slurmdbd: saved 1846 pending RPCs [2016-09-26T08:02:17.024] Unable
> > to remove pidfile '/var/run/slurmctld.pid': Permission denied
> > [2016-09-26T08:02:16.582] killing old slurmctld[20343]
> > [2016-09-26T08:02:17.028] Job accounting information stored, but
> > details not gathered [2016-09-26T08:02:17.028] slurmctld version
> > 2.6.9 started on cluster graph [2016-09-26T08:02:17.029] debug3:
> > Trying to load plugin /usr/local/lib/slurm/crypto_munge.so
> > [2016-09-26T08:02:17.052] Munge cryptographic signature plugin
> > loaded [2016-09-26T08:02:17.052] debug3: Success.
> > [2016-09-26T08:02:17.052] debug3: Trying to load plugin
> > /usr/local/lib/slurm/gres_gpu.so [2016-09-26T08:02:17.060] debug:
> > init: Gres GPU plugin loaded [2016-09-26T08:02:17.060] debug3:
> > Success.
> >
> > No crontab set, anything, it just restarts itself. And the thing
> > is, when I got jobs running for more than 3 days, they are killed
> > by this restart (even if, normally, slurm is capable to resume
> > jobs) :
> >
> > [2016-09-26T08:02:01.282] [830] profile signalling type Task
> > [2016-09-26T08:02:20.632] debug3: in the service_connection
> > [2016-09-26T08:02:20.634] debug2: got this type of message 1001
> > [2016-09-26T08:02:20.634] debug2: Processing RPC:
> > REQUEST_NODE_REGISTRATION_STATUS [2016-09-26T08:02:20.635] debug3:
> > CPUs=22 Boards=1 Sockets=22 Cores=1 Threads=1 Memory=145060
> > TmpDisk=78723 Uptime=4210729 [2016-09-26T08:02:20.636] debug4:
> > found jobid = 830, stepid = 4294967294 [2016-09-26T08:02:20.637]
> > [830] Called _msg_socket_accept [2016-09-26T08:02:20.637] [830]
> > Leaving _msg_socket_accept [2016-09-26T08:02:20.637] [830] eio:
> > handling events for 1 objects [2016-09-26T08:02:20.637] [830]
> > Called _msg_socket_readable [2016-09-26T08:02:20.637] [830]
> > Entering _handle_accept (new thread) [2016-09-26T08:02:20.638]
> > [830] Identity: uid=0, gid=0 [2016-09-26T08:02:20.638] [830]
> > Entering _handle_request [2016-09-26T08:02:20.639] [830] Got
> > request 5 [2016-09-26T08:02:20.639] [830] Handling REQUEST_STATE
> > [2016-09-26T08:02:20.639] [830] Leaving _handle_request:
> > SLURM_SUCCESS [2016-09-26T08:02:20.639] [830] Entering
> > _handle_request [2016-09-26T08:02:20.639] debug: found apparently
> > running job 830 [2016-09-26T08:02:20.640] [830] Leaving
> > _handle_accept [2016-09-26T08:02:20.692] debug3: in the
> > service_connection [2016-09-26T08:02:20.693] debug2: got this type
> > of message 6013 [2016-09-26T08:02:20.693] debug2: Processing RPC:
> > REQUEST_ABORT_JOB [2016-09-26T08:02:20.694] debug: _rpc_abort_job,
> > uid = 64030 [2016-09-26T08:02:20.694] debug:
> > task_slurmd_release_resources: 830 [2016-09-26T08:02:20.694]
> > debug3: state for jobid 830: ctime:1474633733 revoked:0 expires:0
> > [2016-09-26T08:02:20.694] debug3: state for jobid 834:
> > ctime:1474839186 revoked:1474840473 expires:1474840473
> > [2016-09-26T08:02:20.694] debug3: destroying job 834 state
> > [2016-09-26T08:02:20.694] debug3: state for jobid 835:
> > ctime:1474840474 revoked:1474840567 expires:1474840567
> > [2016-09-26T08:02:20.694] debug3: destroying job 835 state
> > [2016-09-26T08:02:20.694] debug3: state for jobid 836:
> > ctime:1474840570 revoked:1474840617 expires:1474840617
> > [2016-09-26T08:02:20.694] debug3: destroying job 836 state
> > [2016-09-26T08:02:20.694] debug3: state for jobid 837:
> > ctime:1474840618 revoked:1474840672 expires:1474840672
> > [2016-09-26T08:02:20.694] debug3: destroying job 837 state
> > [2016-09-26T08:02:20.694] debug3: state for jobid 838:
> > ctime:1474840672 revoked:1474840747 expires:1474840747
> > [2016-09-26T08:02:20.694] debug3: destroying job 838 state
> > [2016-09-26T08:02:20.694] debug3: state for jobid 839:
> > ctime:1474840748 revoked:1474840855 expires:1474840855
> > [2016-09-26T08:02:20.694] debug3: destroying job 839 state
> > [2016-09-26T08:02:20.694] debug3: state for jobid 840:
> > ctime:1474840883 revoked:1474840969 expires:1474840969
> > [2016-09-26T08:02:20.694] debug3: destroying job 840 state
> > [2016-09-26T08:02:20.695] debug: credential for job 830 revoked
> > [2016-09-26T08:02:20.696] debug4: found jobid = 830, stepid =
> > 4294967294 [2016-09-26T08:02:20.699] [830] Called
> > _msg_socket_accept [2016-09-26T08:02:20.699] [830] Leaving
> > _msg_socket_accept [2016-09-26T08:02:20.699] [830] eio: handling
> > events for 1 objects [2016-09-26T08:02:20.699] [830] Called
> > _msg_socket_readable [2016-09-26T08:02:20.700] [830] Entering
> > _handle_accept (new thread) [2016-09-26T08:02:20.700] [830]
> > Identity: uid=0, gid=0 [2016-09-26T08:02:20.701] [830] Entering
> > _handle_request [2016-09-26T08:02:20.701] debug2: container signal
> > 997 to job 830.4294967294 [2016-09-26T08:02:20.701] [830] Got
> > request 4 [2016-09-26T08:02:20.701] [830] Handling
> > REQUEST_SIGNAL_CONTAINER [2016-09-26T08:02:20.701] [830]
> > _handle_signal_container for step=830.4294967294 uid=0 signal=997
> > [2016-09-26T08:02:20.701] [830] Myname in build_hashtbl:
> > (slurmstepd) [2016-09-26T08:02:20.723] [830] Sending signal 9 to
> > pid 649 (python) [2016-09-26T08:02:20.723] [830] Sending signal 9
> > to pid 648 (slurm_script) [2016-09-26T08:02:20.723] [830] Sent
> > signal 9 to 830.4294967294 [2016-09-26T08:02:20.724] debug4: found
> > jobid = 830, stepid = 4294967294 [2016-09-26T08:02:20.724] [830]
> > task 0 (648) exited. Killed by signal 9. [2016-09-26T08:02:20.725]
> > [830] Called _msg_socket_accept [2016-09-26T08:02:20.725] [830]
> > task_post_term: 830.4294967294, task 0 [2016-09-26T08:02:20.725]
> > [830] Aggregated 1 task exit messages [2016-09-26T08:02:20.725]
> > [830] sending task exit msg for 1 tasks [2016-09-26T08:02:20.725]
> > [830] Leaving _msg_socket_accept [2016-09-26T08:02:20.725] [830]
> > eio: handling events for 1 objects [2016-09-26T08:02:20.725] [830]
> > Called _msg_socket_readable [2016-09-26T08:02:20.725] [830] Leaving
> > _handle_request: SLURM_SUCCESS [2016-09-26T08:02:20.726] [830]
> > Entering _handle_request [2016-09-26T08:02:20.726] [830] Leaving
> > _handle_accept [2016-09-26T08:02:20.726] [830] Entering
> > _handle_accept (new thread) [2016-09-26T08:02:20.727] [830] Myname
> > in build_hashtbl: (slurmstepd) [2016-09-26T08:02:20.727] [830]
> > Identity: uid=0, gid=0 [2016-09-26T08:02:20.727] [830] Entering
> > _handle_request [2016-09-26T08:02:20.728] [830] Got request 5
> > [2016-09-26T08:02:20.728] [830] Handling REQUEST_STATE
> > [2016-09-26T08:02:20.728] [830] Leaving _handle_request:
> > SLURM_SUCCESS [2016-09-26T08:02:20.728] [830] Entering
> > _handle_request [2016-09-26T08:02:20.728] [830] Leaving
> > _handle_accept [2016-09-26T08:02:20.744] [830] cpu_freq_reset:
> > #cpus reset = 0 [2016-09-26T08:02:20.744] [830] Before call to
> > spank_fini() [2016-09-26T08:02:20.744] [830] After call to
> > spank_fini() [2016-09-26T08:02:20.744] [830] eio: handling events
> > for 1 objects [2016-09-26T08:02:20.744] [830] Called
> > _msg_socket_readable [2016-09-26T08:02:20.744] [830] false,
> > shutdown [2016-09-26T08:02:20.744] [830] Message thread exited
> > [2016-09-26T08:02:20.745] [830] step 830.4294967294 abort
> > completed [2016-09-26T08:02:20.745] [830] done with job
> >
> >
> >
> > So, if anybody can help me with this issue, then my slurm will
> > works perfectly with GPUs&CPUs :)
> >
> > Thank you very much
> >
> > Regards, Philippe
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQEcBAEBCAAGBQJX6XTPAAoJEEmckBqrs5nBGswIAJlqvvfdq7sOoj4X/H3+YMmQ
> rIJz9/pTjLQj65SSa7fLFCLhyskZeNODmy1du2cdNf6YS2THnK53Ixk5Y1iDl/Gn
> ki+nzeDpd9fccqIs4a51yYQRJiRcX3ZAkwoF+QFooqepMLYgeZE26rhKVrnqr+5r
> E+cI+BPw2C2jsNWQYS61hhRm04mZmIdJ+MGP54aAfrinTR6UBp/FrysGGhzU7RP+
> M0gRGIVM40SqANuJM48Vpa9Qa0z77q6VbMLtHgpx8L+Ge6mlzjgWSEncK0b83vLH
> khulSjpYJiFopsvoNTq1lLjfaOXieOPnF1nppNi7NGHQADh5FyHZo5JCDCQC1AU=
> =Z4ro
> -----END PGP SIGNATURE-----
>
