-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Philippe,
> If I can't use logrotate, what must I use ? I just disabled it, I'm
> gonna see if the problem still persist.
You can use logrotate. I'd suggest using a much larger "size" though.
For example, we don't rotate the logs until at least 2GB.
> * My SlurmctldTimeout is set to 300, in the doc it seems like it's
> in seconds, so 300 seconds for a timeout is way enough I hope !
You're correct, it's in seconds; 5 minutes should be more than enough
for slurmctld to restart.
> * If there's typos, slurm doesn't warn for them at start or
> somewhere later in the logs ?
If there are typos and/or invalid syntax in slurm.conf, slurmctld will
not start - at least that's what I've seen.
> Thanks for reply, if you have any other idea, don't hesitate to
> share !
What I would do is perform a restart of slurm using the "postrotate"
command below, but remove the "--quiet" and ">/dev/null", and prefix
"time" to it, e.g.:
time /usr/sbin/invoke-rc.d slurm-llnl reconfig
This way you'll be able to verify how long a restart takes, and output
will be dumped to the screen.
HTH,
John DeSantis
On 09/27/2016 03:38 AM, Philippe wrote:
> Hi John, thanks for the reply :)
>
> Yes I've got logrotate enabled for my slurm :
> /var/log/slurm/slurmd.log /var/log/slurm/slurmctld.log
> /var/log/slurm/slurmdbd.log { compress missingok nocopytruncate
> nocreate nodelaycompress nomail notifempty noolddir rotate 12
> sharedscripts size=5M postrotate /usr/sbin/invoke-rc.d --quiet
> slurm-llnl reconfig >/dev/null endscript }
>
> If I can't use logrotate, what must I use ? I just disabled it, I'm
> gonna see if the problem still persist.
>
> * My SlurmctldTimeout is set to 300, in the doc it seems like it's
> in seconds, so 300 seconds for a timeout is way enough I hope ! *
> If there's typos, slurm doesn't warn for them at start or somewhere
> later in the logs ? * No upgrade since the compilation.
>
> Thanks for reply, if you have any other idea, don't hesitate to
> share !
>
> Regards,
>
>
> On Mon, Sep 26, 2016 at 9:39 PM, John DeSantis <[email protected]>
> wrote:
>
>>
> Philippe,
>
>>>> But every 3 days, the slurmctld process restarts by itself,
>>>> as you can see in slurmctld.log :
>
> ... SNIP ...
>
>>>> No crontab set, anything, it just restarts itself. And the
>>>> thing is, when I got jobs running for more than 3 days, they
>>>> are killed by this restart (even if, normally, slurm is
>>>> capable to resume jobs)
>
> Do you have log rotation enabled that is stopping and restarting
> the ctl d?
>
> As far as the lost jobs go, check your 'SlurmctldTimeout' and see
> if it's set too low. We've never lost any jobs due to:
>
> * ctld restarts * typos in slurm.conf (!!) * upgrades
>
> I've been especially guilty of typos, and FWIW SLURM has been
> extremely forgiving.
>
> HTH, John DeSantis
>
>
>
> On 09/26/2016 03:46 AM, Philippe wrote:
>>>> Hello everybody, I'm trying to understand an issue with 2
>>>> SLURM installations on Ubuntu 14.04 64b, with slurm 2.6.9
>>>> compiled. Only 1 computer running Slurmctld/slurmd/slurmdbd.
>>>> It works very well for any job that doesn't last more than 2
>>>> days.
>>>>
>>>> But every 3 days, the slurmctld process restarts by itself,
>>>> as you can see in slurmctld.log :
>>>>
>>>> [2016-09-26T08:01:42.682] debug2: Performing purge of old
>>>> job records [2016-09-26T08:01:42.683] debug: sched: Running
>>>> job scheduler [2016-09-26T08:01:42.683] debug2: Performing
>>>> full system state save [2016-09-26T08:01:44.003] debug:
>>>> slurmdbd: DBD_RC is -1 from DBD_FLUSH_JOBS(1408): (null)
>>>> [2016-09-26T08:01:44.751] debug2: Sending cpu count of 22 for
>>>> cluster [2016-09-26T08:01:44.792] debug: slurmdbd: Issue
>>>> with call DBD_CLUSTER_CPUS(1407): 4294967295(This cluster
>>>> hasn't been added to accounting yet)
>>>> [2016-09-26T08:01:46.792] debug2: Testing job time limits
>>>> and checkpoints [2016-09-26T08:01:47.005] debug3: Writing job
>>>> id 840 to header record of job_state file
>>>> [2016-09-26T08:02:03.003] debug: slurmdbd: DBD_RC is -1 from
>>>> DBD_FLUSH_JOBS(1408): (null) [2016-09-26T08:02:16.582]
>>>> Terminate signal (SIGINT or SIGTERM) received
>>>> [2016-09-26T08:02:16.582] debug: sched: slurmctld
>>>> terminating [2016-09-26T08:02:16.583] debug3:
>>>> _slurmctld_rpc_mgr shutting down [2016-09-26T08:02:16.798]
>>>> Saving all slurm state [2016-09-26T08:02:16.801] debug3:
>>>> Writing job id 840 to header record of job_state file
>>>> [2016-09-26T08:02:16.881] debug3: _slurmctld_background
>>>> shutting down [2016-09-26T08:02:16.952] slurmdbd: saved 1846
>>>> pending RPCs [2016-09-26T08:02:17.024] Unable to remove
>>>> pidfile '/var/run/slurmctld.pid': Permission denied
>>>> [2016-09-26T08:02:16.582] killing old slurmctld[20343]
>>>> [2016-09-26T08:02:17.028] Job accounting information stored,
>>>> but details not gathered [2016-09-26T08:02:17.028] slurmctld
>>>> version 2.6.9 started on cluster graph
>>>> [2016-09-26T08:02:17.029] debug3: Trying to load plugin
>>>> /usr/local/lib/slurm/crypto_munge.so
>>>> [2016-09-26T08:02:17.052] Munge cryptographic signature
>>>> plugin loaded [2016-09-26T08:02:17.052] debug3: Success.
>>>> [2016-09-26T08:02:17.052] debug3: Trying to load plugin
>>>> /usr/local/lib/slurm/gres_gpu.so [2016-09-26T08:02:17.060]
>>>> debug: init: Gres GPU plugin loaded [2016-09-26T08:02:17.060]
>>>> debug3: Success.
>>>>
>>>> No crontab set, anything, it just restarts itself. And the
>>>> thing is, when I got jobs running for more than 3 days, they
>>>> are killed by this restart (even if, normally, slurm is
>>>> capable to resume jobs) :
>>>>
>>>> [2016-09-26T08:02:01.282] [830] profile signalling type Task
>>>> [2016-09-26T08:02:20.632] debug3: in the service_connection
>>>> [2016-09-26T08:02:20.634] debug2: got this type of message
>>>> 1001 [2016-09-26T08:02:20.634] debug2: Processing RPC:
>>>> REQUEST_NODE_REGISTRATION_STATUS [2016-09-26T08:02:20.635]
>>>> debug3: CPUs=22 Boards=1 Sockets=22 Cores=1 Threads=1
>>>> Memory=145060 TmpDisk=78723 Uptime=4210729
>>>> [2016-09-26T08:02:20.636] debug4: found jobid = 830, stepid =
>>>> 4294967294 [2016-09-26T08:02:20.637] [830] Called
>>>> _msg_socket_accept [2016-09-26T08:02:20.637] [830] Leaving
>>>> _msg_socket_accept [2016-09-26T08:02:20.637] [830] eio:
>>>> handling events for 1 objects [2016-09-26T08:02:20.637]
>>>> [830] Called _msg_socket_readable [2016-09-26T08:02:20.637]
>>>> [830] Entering _handle_accept (new thread)
>>>> [2016-09-26T08:02:20.638] [830] Identity: uid=0, gid=0
>>>> [2016-09-26T08:02:20.638] [830] Entering _handle_request
>>>> [2016-09-26T08:02:20.639] [830] Got request 5
>>>> [2016-09-26T08:02:20.639] [830] Handling REQUEST_STATE
>>>> [2016-09-26T08:02:20.639] [830] Leaving _handle_request:
>>>> SLURM_SUCCESS [2016-09-26T08:02:20.639] [830] Entering
>>>> _handle_request [2016-09-26T08:02:20.639] debug: found
>>>> apparently running job 830 [2016-09-26T08:02:20.640] [830]
>>>> Leaving _handle_accept [2016-09-26T08:02:20.692] debug3: in
>>>> the service_connection [2016-09-26T08:02:20.693] debug2: got
>>>> this type of message 6013 [2016-09-26T08:02:20.693] debug2:
>>>> Processing RPC: REQUEST_ABORT_JOB [2016-09-26T08:02:20.694]
>>>> debug: _rpc_abort_job, uid = 64030 [2016-09-26T08:02:20.694]
>>>> debug: task_slurmd_release_resources: 830
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 830:
>>>> ctime:1474633733 revoked:0 expires:0
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 834:
>>>> ctime:1474839186 revoked:1474840473 expires:1474840473
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 834 state
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 835:
>>>> ctime:1474840474 revoked:1474840567 expires:1474840567
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 835 state
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 836:
>>>> ctime:1474840570 revoked:1474840617 expires:1474840617
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 836 state
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 837:
>>>> ctime:1474840618 revoked:1474840672 expires:1474840672
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 837 state
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 838:
>>>> ctime:1474840672 revoked:1474840747 expires:1474840747
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 838 state
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 839:
>>>> ctime:1474840748 revoked:1474840855 expires:1474840855
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 839 state
>>>> [2016-09-26T08:02:20.694] debug3: state for jobid 840:
>>>> ctime:1474840883 revoked:1474840969 expires:1474840969
>>>> [2016-09-26T08:02:20.694] debug3: destroying job 840 state
>>>> [2016-09-26T08:02:20.695] debug: credential for job 830
>>>> revoked [2016-09-26T08:02:20.696] debug4: found jobid = 830,
>>>> stepid = 4294967294 [2016-09-26T08:02:20.699] [830] Called
>>>> _msg_socket_accept [2016-09-26T08:02:20.699] [830] Leaving
>>>> _msg_socket_accept [2016-09-26T08:02:20.699] [830] eio:
>>>> handling events for 1 objects [2016-09-26T08:02:20.699] [830]
>>>> Called _msg_socket_readable [2016-09-26T08:02:20.700] [830]
>>>> Entering _handle_accept (new thread)
>>>> [2016-09-26T08:02:20.700] [830] Identity: uid=0, gid=0
>>>> [2016-09-26T08:02:20.701] [830] Entering _handle_request
>>>> [2016-09-26T08:02:20.701] debug2: container signal 997 to job
>>>> 830.4294967294 [2016-09-26T08:02:20.701] [830] Got request 4
>>>> [2016-09-26T08:02:20.701] [830] Handling
>>>> REQUEST_SIGNAL_CONTAINER [2016-09-26T08:02:20.701] [830]
>>>> _handle_signal_container for step=830.4294967294 uid=0
>>>> signal=997 [2016-09-26T08:02:20.701] [830] Myname in
>>>> build_hashtbl: (slurmstepd) [2016-09-26T08:02:20.723] [830]
>>>> Sending signal 9 to pid 649 (python)
>>>> [2016-09-26T08:02:20.723] [830] Sending signal 9 to pid 648
>>>> (slurm_script) [2016-09-26T08:02:20.723] [830] Sent signal 9
>>>> to 830.4294967294 [2016-09-26T08:02:20.724] debug4: found
>>>> jobid = 830, stepid = 4294967294 [2016-09-26T08:02:20.724]
>>>> [830] task 0 (648) exited. Killed by signal 9.
>>>> [2016-09-26T08:02:20.725] [830] Called _msg_socket_accept
>>>> [2016-09-26T08:02:20.725] [830] task_post_term:
>>>> 830.4294967294, task 0 [2016-09-26T08:02:20.725] [830]
>>>> Aggregated 1 task exit messages [2016-09-26T08:02:20.725]
>>>> [830] sending task exit msg for 1 tasks
>>>> [2016-09-26T08:02:20.725] [830] Leaving _msg_socket_accept
>>>> [2016-09-26T08:02:20.725] [830] eio: handling events for 1
>>>> objects [2016-09-26T08:02:20.725] [830] Called
>>>> _msg_socket_readable [2016-09-26T08:02:20.725] [830] Leaving
>>>> _handle_request: SLURM_SUCCESS [2016-09-26T08:02:20.726]
>>>> [830] Entering _handle_request [2016-09-26T08:02:20.726]
>>>> [830] Leaving _handle_accept [2016-09-26T08:02:20.726] [830]
>>>> Entering _handle_accept (new thread)
>>>> [2016-09-26T08:02:20.727] [830] Myname in build_hashtbl:
>>>> (slurmstepd) [2016-09-26T08:02:20.727] [830] Identity: uid=0,
>>>> gid=0 [2016-09-26T08:02:20.727] [830] Entering
>>>> _handle_request [2016-09-26T08:02:20.728] [830] Got request
>>>> 5 [2016-09-26T08:02:20.728] [830] Handling REQUEST_STATE
>>>> [2016-09-26T08:02:20.728] [830] Leaving _handle_request:
>>>> SLURM_SUCCESS [2016-09-26T08:02:20.728] [830] Entering
>>>> _handle_request [2016-09-26T08:02:20.728] [830] Leaving
>>>> _handle_accept [2016-09-26T08:02:20.744] [830]
>>>> cpu_freq_reset: #cpus reset = 0 [2016-09-26T08:02:20.744]
>>>> [830] Before call to spank_fini() [2016-09-26T08:02:20.744]
>>>> [830] After call to spank_fini() [2016-09-26T08:02:20.744]
>>>> [830] eio: handling events for 1 objects
>>>> [2016-09-26T08:02:20.744] [830] Called _msg_socket_readable
>>>> [2016-09-26T08:02:20.744] [830] false, shutdown
>>>> [2016-09-26T08:02:20.744] [830] Message thread exited
>>>> [2016-09-26T08:02:20.745] [830] step 830.4294967294 abort
>>>> completed [2016-09-26T08:02:20.745] [830] done with job
>>>>
>>>>
>>>>
>>>> So, if anybody can help me with this issue, then my slurm
>>>> will works perfectly with GPUs&CPUs :)
>>>>
>>>> Thank you very much
>>>>
>>>> Regards, Philippe
>>>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJX6mt2AAoJEEmckBqrs5nB20wH/14Gt4t8xA0bQfbwZwItwxCk
JKE93apRJ7UXnCbLNl9QkmJW+PgwzMi/86VqoHBC+fMlDVQk7UtwTzXmnT1GFN7y
DcfwHHb33S53HiyZ1LM5i6Bm6troUHK127sCx4LP2IU96QNRBMz03p2GdDyA6Ct0
XAZSAzCuaUpwzaAnhOOnge2Rxrivi/qBHklv9bRDEPINYB9B3vEDJfEa+9f9fJd8
+d+DXUTNBiIormXfVvAyGgfwSNFTcpN2BZiH8Hju452GLdbuTpP5DVAvmATtqypY
n4ZYEH8ujm021fBkvvmgL3fReNOMqYO1ZIiPqnU47wEYq7rlZxBt9LOfGlYm9Pk=
=hHwL
-----END PGP SIGNATURE-----
