Hi,
Running 15.08.3. Life is lovely for most users but one group runs array jobs,
100 tasks per array, whose tasks are in the 30-45 second range. Though
allocated 1800 cores they seldom get beyond 900 as slurmctld is unable to keep
up it seems. On nodes we see "Retrying job complete RPC for
nnnnnnn.4294967294" and "slurm_receive_msg: Socket timed out on send/recv
operation". Also, occasionally have nodes go into drain with an error "error:
Job nnnnn already running, do not launch second copy"
squeue is virtually impossible to execute in a timely fashion. Other commands
slow but not as bad.
sdiag often shows 256 server threads, never more, and a queue length of 86K.
Have attached a cut of the sdiag showing the RPC times.
strace of slurmctld yields frequent "futex(0x886584, FUTEX_WAIT_PRIVATE,
13128476, NULL) = -1 EAGAIN (Resource temporarily unavailable)"
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
53.58 1.655625 568 2915 937 futex
46.34 1.431783 5486 261 nanosleep
Have followed the suggestions in the "High Computing Guide". The server itself
is not memory/disk IO bound or indicating any OS/HW performance issues. Just
100-130% CPU use for slurmctld.
I suspect that increasing the MAX_SERVER_THREAD would mitigate the failed RPC
calls. Does this sound correct? Can this be changed dynamically without a
recompile?
Suggestions welcome.
Thank you,
Doug Meyer
Remote Procedure Call statistics by message type
REQUEST_COMPLETE_BATCH_SCRIPT ( 5018) count:456871
ave_time:11255252 total_time:5142198651289
MESSAGE_EPILOG_COMPLETE ( 6012) count:107888
ave_time:2110978 total_time:227749298044
REQUEST_SUBMIT_BATCH_JOB ( 4003) count:22972
ave_time:753299 total_time:17304793369
MESSAGE_NODE_REGISTRATION_STATUS ( 1002) count:3621
ave_time:312952 total_time:1133200718
REQUEST_NODE_INFO ( 2007) count:231
ave_time:141062 total_time:32585420
REQUEST_PARTITION_INFO ( 2009) count:120
ave_time:363 total_time:43634
REQUEST_JOB_REQUEUE ( 5023) count:25
ave_time:112283 total_time:2807078
REQUEST_JOB_INFO ( 2003) count:24
ave_time:3997254 total_time:95934099
REQUEST_JOB_USER_INFO ( 2039) count:17
ave_time:1084437 total_time:18435430
REQUEST_JOB_ALLOCATION_INFO_LITE ( 4016) count:16
ave_time:718915 total_time:11502653
REQUEST_JOB_STEP_CREATE ( 5001) count:16
ave_time:176642 total_time:2826286
REQUEST_UPDATE_NODE ( 3002) count:15
ave_time:154750 total_time:2321257
REQUEST_STEP_COMPLETE ( 5016) count:14
ave_time:223278 total_time:3125905
REQUEST_CANCEL_JOB_STEP ( 5005) count:14
ave_time:340936 total_time:4773111
REQUEST_KILL_JOB ( 5032) count:9
ave_time:250563 total_time:2255070
REQUEST_STATS_INFO ( 2035) count:6
ave_time:858 total_time:5153
REQUEST_NODE_INFO_SINGLE ( 2040) count:4
ave_time:122114 total_time:488459
REQUEST_JOB_INFO_SINGLE ( 2021) count:2
ave_time:2747572 total_time:5495144
ACCOUNTING_REGISTER_CTLD (10003) count:1
ave_time:80731 total_time:80731
