Sorry for the late response, I wasn't at work. I think this will be difficult to solve this problems if you are not the admin of the cluster. The error below is probably not linked to the one you have sent before.
I would suggest to run the job in debug mode to see if you get anything else in the error message. It could be that you are hitting a linux system limit, such as max open files, somaxconn or backlog. See:https://slurm.schedmd.com/big_sys.html Cheers, Barbara On 01/17/2017 09:21 AM, Nigella Sanders wrote: > Re: [slurm-dev] Re: Meaning of error messages from sbatch > > Thank you for your reply Barbara, > > I have already seen that the errors I mentioned above are mostly > harmless and do not usually affect to executions times or computation > results. However I guess they appear for some reason and I would like > if are due to the code or the cluster software/hardware itself. > > Anyway, even simple single-node task-farming jobs like mine produce > sometimes serious errors like: > > srun: error: _server_read: fd 12 n 0 got error or unexpected eof > reading header: No error > srun: error: step_launch_notify_io_failure: aborting, io error with > slurmstepd on node 0 > srun: Job step aborted: Waiting up to 32 seconds for job step to finish. > srun: error: Timed out waiting for job step to complete > > which end up with step cancellation. > > > P.D.: unfortunately I don't access to IB diagnostic tools. > > > Regards, > Nigella > > > > > > > > > 2017-01-16 12:41 GMT+01:00 Barbara Krašovec <[email protected] > <mailto:[email protected]>>: > > Hello! > > > On 01/13/2017 08:18 AM, Nigella Sanders wrote: >> >> >> Hi all, >> >> >> When launching my simple task farming jobs (described here >> <https://groups.google.com/forum/#%21topic/slurm-devel/SMG2RvKY7Qc>), >> sbatch usually reports errors like these: >> >> * ibwarn: [100226] _do_madrpc: recv failed: Interrupted system call >> ibwarn: [100226] mad_rpc: _do_madrpc failed; dport (Lid 150) >> slurmstepd: ofed: No error >> ibwarn: [12463] _do_madrpc: recv failed: Interrupted system call >> ibwarn: [12463] mad_rpc: _do_madrpc failed; dport (Lid 114) >> slurmstepd: classportinfo query: No error* >> >> What do they mean? >> I guess they are related to infiniband I but don't know how to >> manage them or what to look into to fix them. >> Most of times they don't affect the results and jobs complete ok. >> >> Any clue on this would be much appreciated, >> >> Regards, >> Nigella. > I wouldn't worry too much, if the interrupted syscalls do not turn > into errors (for now you only see warnings). I would try to run > some diagnostic with ibdiagnet and try ibping.. > If the warnings are present on every node, maybe you should > consider a firmware upgrade. > > Cheers, > Barbara > >
