Ralph, indeed. As I have said before: Finally, my one piece of advice to everyone managing batch systems. It is a name resolution problem. No, really it is. Even if your cluster catches fire, the real reason that your jobs are not being submitted is that the DNS resolver is burning and the scheduler can't resolve the hostname of the submit host.
I am having some problems with a GPFS cluster today. Guess what - name resolution. We have some new hosts which for a reason have multiple IP addresses. Name resolution is producing 'fun and games'. On 25 October 2017 at 17:30, [email protected] <[email protected]> wrote: > Good points. I would also caution against renaming nodes using interfaces. > This frequently causes failure of 3rd party software packages that compare > the return value of “hostname” to the list of allocated nodes for > optimization or placement purposes - e.g., mpirun! A quick grep of the > mailing list logs will reveal all the woes that created. > > On Oct 25, 2017, at 8:22 AM, John Hearns <[email protected]> wrote: > > When using “mpirun” we can specify “-iface ib0” this is true, and the > exact syntax depends on your MPI of choice, as noted above. > > However, don't get confused between IPOIB and Infiniband itself. IPOIB > is of course sending IP traffic over Infiniband. > An Infiniband network can perfectly happily function without any IP > addresses being assigned. > The point I am getting at is that the IP connections should be used to set > up/launch the job, depending on the launcher used by that MPI (eg Hydra as > above) > What I am saying is do not get confused between the activity which sets up > the MPI processes on the remote nodes, and the actual MPI traffic. > > Unless you really want to use IPOIB for your MPI traffic (maybe for doing > a benchmark comparison) I would say just run the srun with the ethernet > IPs and let your MPI choose > the best bit transport layer (BTL in OpenMPI speak). > > What I would do is tag the Infiniband equipped nodes with a feature called > 'IB' or 'nonIB' for the others, and choose those nodes. > (Sorry - my head is in PBSPro world these days so that would be a > resources_available in that world) > > My advice - schedule just on the IB equipped nodes. Run your mpi with a > verbose flag and see which BTL it is choosing. > You may be pleasantly surprised! > I would say 'take down the ib0 interface' but that may be a bad move - it > is probably used for storage mounts at least. > > > If I have misunderstood the point, and have been a bit rude here I > apologise in advance. Someone with a clue will come along > and slap me rounfd the head I am sure. > > > > > > > > > > > > > > > > > > > > > > On 25 October 2017 at 17:03, Le Biot, Pierre-Marie < > [email protected]> wrote: > >> Hi Sebastian, >> >> >> >> Another solution could be to change the configuration of nodes in >> slurm.conf, making use of NodeName and NodeHostname (and NodeAddr if >> needed) : >> >> >> >> “ >> >> NodeName >> >> Name that Slurm uses to refer to a node[...]. Typically this would be the >> string that "/bin/hostname -s" returns.[...]It may also be an arbitrary >> string if NodeHostname is specified.[...] >> >> >> >> NodeHostname >> >> Typically this would be the string that "/bin/hostname -s" >> returns.[...]By default, the NodeHostname will be identical in value to >> NodeName. >> >> >> >> NodeAddr >> >> Name that a node should be referred to in establishing a communications >> path.[...] NodeAddr may also contain IP addresses. By default, the NodeAddr >> will be identical in value to NodeHostname. >> >> “ >> >> >> >> For the nodes having an infiniband interface declare the associated name >> in NodeName and the regular hostname in NodeHostname. >> >> SLURM_NODELIST will contain the names declared in NodeName. >> >> >> >> Regards, >> >> Pierre-Marie Le Biot >> >> >> >> *From:* Sebastian Eastham [mailto:[email protected]] >> *Sent:* Tuesday, October 24, 2017 10:02 PM >> *To:* slurm-dev <[email protected]> >> *Subject:* [slurm-dev] Selecting a network interface with srun >> >> >> >> Dear Slurm Developers mailing list, >> >> >> >> When calling the “srun” command, is there any way to specify the desired >> network interface? Our network is a mix of ethernet and inifiniband, such >> that only a subset of the nodes have an infiniband interface. When using >> “mpirun” we can specify “-iface ib0”, but there does not appear to be a >> similar option for “srun”. Although we can successfully run our >> applications with “srun”, we can see from “iftop” that the application is >> communicating purely through the ethernet interface. >> >> >> >> Once again, I appreciate any help or guidance that you can give me! >> >> >> >> Regards, >> >> >> >> Seb >> >> >> >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> >> Dr. Sebastian D. Eastham >> >> Research Scientist >> >> Laboratory for Aviation and the Environment >> >> Massachusetts Institute of Technology >> >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= >> >> >> >> >> > > >
