Hi Ralph,
I have completed rebuild of openmpi1.7rc8. To save time, I added --disable-vt. ( Is it OK? ) Well, what shall I do ? ./configure \ --prefix=/home/mishima/opt/mpi/openmpi-1.7rc8-pgi12.9 \ --with-tm \ --with-verbs \ --disable-ipv6 \ --disable-vt \ --enable-debug \ CC=pgcc CFLAGS="-fast -tp k8-64e" \ CXX=pgCC CXXFLAGS="-fast -tp k8-64e" \ F77=pgfortran FFLAGS="-fast -tp k8-64e" \ FC=pgfortran FCFLAGS="-fast -tp k8-64e" Note: I tried patch user.diff after rebuiding openmpi1.7rc8. But, I got an error and could not go foward. $ patch -p0 < user.diff # this is OK $ make # I got an error CC util/hostfile/hostfile.lo PGC-S-0037-Syntax error: Recovery attempted by deleting <string> (util/hostfile/hostfile.c: 728) PGC/x86-64 Linux 12.9-0: compilation completed with severe errors Regards, Tetsuya Mishima > Could you please apply the attached patch and try it again? If you haven't had time to configure with --enable-debug, that is fine - this will output regardless. > > Thanks > Ralph > > - user.diff > > > On Mar 20, 2013, at 4:59 PM, Ralph Castain <r...@open-mpi.org> wrote: > > > You obviously have some MCA params set somewhere: > > > >> -------------------------------------------------------------------------- > >> A deprecated MCA parameter value was specified in an MCA parameter > >> file. Deprecated MCA parameters should be avoided; they may disappear > >> in future releases. > >> > >> Deprecated parameter: orte_rsh_agent > >> -------------------------------------------------------------------------- > > > > Check your environment for anything with OMPI_MCA_xxx, and your default MCA parameter file to see what has been specified. > > > > The allocation looks okay - I'll have to look for other debug flags you can set. Meantime, can you please add --enable-debug to your configure cmd line and rebuild? > > > > Thanks > > Ralph > > > > > > On Mar 20, 2013, at 4:39 PM, tmish...@jcity.maeda.co.jp wrote: > > > >> > >> > >> Hi Ralph, > >> > >> Here is a result of rerun with --display-allocation. > >> I set OMP_NUM_THREADS=1 to make the problem clear. > >> > >> Regards, > >> Tetsuya Mishima > >> > >> P.S. As far as I checked, these 2 cases are OK(no problem). > >> (1)mpirun -v -np $NPROCS-x OMP_NUM_THREADS --display-allocation > >> ~/Ducom/testbed/mPre m02-ld > >> (2)mpirun -v -x OMP_NUM_THREADS --display-allocation ~/Ducom/testbed/mPre > >> m02-ld > >> > >> Script File: > >> > >> #!/bin/sh > >> #PBS -A tmishima > >> #PBS -N Ducom-run > >> #PBS -j oe > >> #PBS -l nodes=2:ppn=4 > >> export OMP_NUM_THREADS=1 > >> cd $PBS_O_WORKDIR > >> cp $PBS_NODEFILE pbs_hosts > >> NPROCS=`wc -l < pbs_hosts` > >> mpirun -v -np $NPROCS -hostfile pbs_hosts -x OMP_NUM_THREADS > >> --display-allocation ~/Ducom/testbed/mPre m02-ld > >> > >> Output: > >> -------------------------------------------------------------------------- > >> A deprecated MCA parameter value was specified in an MCA parameter > >> file. Deprecated MCA parameters should be avoided; they may disappear > >> in future releases. > >> > >> Deprecated parameter: orte_rsh_agent > >> -------------------------------------------------------------------------- > >> > >> ====================== ALLOCATED NODES ====================== > >> > >> Data for node: node06 Num slots: 4 Max slots: 0 > >> Data for node: node05 Num slots: 4 Max slots: 0 > >> > >> ================================================================= > >> -------------------------------------------------------------------------- > >> A hostfile was provided that contains at least one node not > >> present in the allocation: > >> > >> hostfile: pbs_hosts > >> node: node06 > >> > >> If you are operating in a resource-managed environment, then only > >> nodes that are in the allocation can be used in the hostfile. You > >> may find relative node syntax to be a useful alternative to > >> specifying absolute node names see the orte_hosts man page for > >> further information. > >> -------------------------------------------------------------------------- > >> > >> > >>> I've submitted a patch to fix the Torque launch issue - just some > >> leftover garbage that existed at the time of the 1.7.0 branch and didn't > >> get removed. > >>> > >>> For the hostfile issue, I'm stumped as I can't see how the problem would > >> come about. Could you please rerun your original test and add > >> "--display-allocation" to your cmd line? Let's see if it is > >>> correctly finding the original allocation. > >>> > >>> Thanks > >>> Ralph > >>> > >>> On Mar 19, 2013, at 5:08 PM, tmish...@jcity.maeda.co.jp wrote: > >>> > >>>> > >>>> > >>>> Hi Gus, > >>>> > >>>> Thank you for your comments. I understand your advice. > >>>> Our script used to be --npernode type as well. > >>>> > >>>> As I told before, our cluster consists of nodes having 4, 8, > >>>> and 32 cores, although it used to be homogeneous at the > >>>> starting time. Furthermore, since performance of each core > >>>> is almost same, a mixed use of nodes with different number > >>>> of cores is possible, just like #PBS -l nodes=1:ppn=32+4:ppn=8. > >>>> > >>>> --npernode type is not applicable to such a mixed use. > >>>> That's why I'd like to continue to use modified hostfile. > >>>> > >>>> By the way, the problem I reported to Jeff yesterday > >>>> was that openmpi-1.7 with torque is something wrong, > >>>> because it caused error against such a simple case as > >>>> shown below, which surprised me. Now, the problem is not > >>>> limited to modified hostfile, I guess. > >>>> > >>>> #PBS -l nodes=4:ppn=8 > >>>> mpirun -np 8 ./my_program > >>>> (OMP_NUM_THREADS=4) > >>>> > >>>> Regards, > >>>> Tetsuya Mishima > >>>> > >>>>> Hi Tetsuya > >>>>> > >>>>> Your script that edits $PBS_NODEFILE into a separate hostfile > >>>>> is very similar to some that I used here for > >>>>> hybrid OpenMP+MPI programs on older versions of OMPI. > >>>>> I haven't tried this in 1.6.X, > >>>>> but it looks like you did and it works also. > >>>>> I haven't tried 1.7 either. > >>>>> Since we run production machines, > >>>>> I try to stick to the stable versions of OMPI (even numbered: > >>>>> 1.6.X, 1.4.X, 1.2.X). > >>>>> > >>>>> I believe you can get the same effect even if you > >>>>> don't edit your $PBS_NODEFILE and let OMPI use it as is. > >>>>> Say, if you choose carefully the values in your > >>>>> #PBS -l nodes=?:ppn=? > >>>>> of your > >>>>> $OMP_NUM_THREADS > >>>>> and use an mpiexec with --npernode or --cpus-per-proc. > >>>>> > >>>>> For instance, for twelve MPI processes, with two threads each, > >>>>> on nodes with eight cores each, I would try > >>>>> (but I haven't tried!): > >>>>> > >>>>> #PBS -l nodes=3:ppn=8 > >>>>> > >>>>> export $OMP_NUM_THREADS=2 > >>>>> > >>>>> mpiexec -np 12 -npernode 4 > >>>>> > >>>>> or perhaps more tightly: > >>>>> > >>>>> mpiexec -np 12 --report-bindings --bind-to-core --cpus-per-proc 2 > >>>>> > >>>>> I hope this helps, > >>>>> Gus Correa > >>>>> > >>>>> > >>>>> > >>>>> On 03/19/2013 03:12 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>> > >>>>>> > >>>>>> Hi Reuti and Gus, > >>>>>> > >>>>>> Thank you for your comments. > >>>>>> > >>>>>> Our cluster is a little bit heterogeneous, which has nodes with 4, 8, > >>>> 32 > >>>>>> cores. > >>>>>> I used 8-core nodes for "-l nodes=4:ppn=8" and 4-core nodes for "-l > >>>>>> nodes=2:ppn=4". > >>>>>> (strictly speaking, Torque picked up proper nodes.) > >>>>>> > >>>>>> As I mentioned before, I usually use openmpi-1.6.x, which has no > >> troble > >>>>>> against that kind > >>>>>> of use. I encountered the issue when I was evaluating openmpi-1.7 to > >>>> check > >>>>>> when we could > >>>>>> move on to it, although we have no positive reason to do that at this > >>>>>> moment. > >>>>>> > >>>>>> As Gus pointed out, I use a script file as shown below for a > >> practical > >>>> use > >>>>>> of openmpi-1.6.x. > >>>>>> > >>>>>> #PBS -l nodes=2:ppn=32 # even "-l nodes=1:ppn=32+4:ppn=8" works fine > >>>>>> export OMP_NUM_THREADS=4 > >>>>>> modify $PBS_NODEFILE pbs_hosts # 64 lines are condensed to 16 lines > >>>> here > >>>>>> mpirun -hostfile pbs_hosts -np 16 -cpus-per-proc 4 -report-bindings \ > >>>>>> -x OMP_NUM_THREADS ./my_program # 32-core node has 8 numanodes, > >> 8-core > >>>>>> node has 2 numanodes > >>>>>> > >>>>>> It works well under the combination of openmpi-1.6.x and Torque. The > >>>>>> problem is just > >>>>>> openmpi-1.7's behavior. > >>>>>> > >>>>>> Regards, > >>>>>> Tetsuya Mishima > >>>>>> > >>>>>>> Hi Tetsuya Mishima > >>>>>>> > >>>>>>> Mpiexec offers you a number of possibilities that you could try: > >>>>>>> --bynode, > >>>>>>> --pernode, > >>>>>>> --npernode, > >>>>>>> --bysocket, > >>>>>>> --bycore, > >>>>>>> --cpus-per-proc, > >>>>>>> --cpus-per-rank, > >>>>>>> --rankfile > >>>>>>> and more. > >>>>>>> > >>>>>>> Most likely one or more of them will fit your needs. > >>>>>>> > >>>>>>> There are also associated flags to bind processes to cores, > >>>>>>> to sockets, etc, to report the bindings, and so on. > >>>>>>> > >>>>>>> Check the mpiexec man page for details. > >>>>>>> > >>>>>>> Nevertheless, I am surprised that modifying the > >>>>>>> $PBS_NODEFILE doesn't work for you in OMPI 1.7. > >>>>>>> I have done this many times in older versions of OMPI. > >>>>>>> > >>>>>>> Would it work for you to go back to the stable OMPI 1.6.X, > >>>>>>> or does it lack any special feature that you need? > >>>>>>> > >>>>>>> I hope this helps, > >>>>>>> Gus Correa > >>>>>>> > >>>>>>> On 03/19/2013 03:00 AM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi Jeff, > >>>>>>>> > >>>>>>>> I didn't have much time to test this morning. So, I checked it > >> again > >>>>>>>> now. Then, the trouble seems to depend on the number of nodes to > >> use. > >>>>>>>> > >>>>>>>> This works(nodes< 4): > >>>>>>>> mpiexec -bynode -np 4 ./my_program&& #PBS -l nodes=2:ppn=8 > >>>>>>>> (OMP_NUM_THREADS=4) > >>>>>>>> > >>>>>>>> This causes error(nodes>= 4): > >>>>>>>> mpiexec -bynode -np 8 ./my_program&& #PBS -l nodes=4:ppn=8 > >>>>>>>> (OMP_NUM_THREADS=4) > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Tetsuya Mishima > >>>>>>>> > >>>>>>>>> Oy; that's weird. > >>>>>>>>> > >>>>>>>>> I'm afraid we're going to have to wait for Ralph to answer why > >> that > >>>> is > >>>>>>>> happening -- sorry! > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mar 18, 2013, at 4:45 PM,<tmish...@jcity.maeda.co.jp> wrote: > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hi Correa and Jeff, > >>>>>>>>>> > >>>>>>>>>> Thank you for your comments. I quickly checked your suggestion. > >>>>>>>>>> > >>>>>>>>>> As a result, my simple example case worked well. > >>>>>>>>>> export OMP_NUM_THREADS=4 > >>>>>>>>>> mpiexec -bynode -np 2 ./my_program&& #PBS -l nodes=2:ppn=4 > >>>>>>>>>> > >>>>>>>>>> But, practical case that more than 1 process was allocated to a > >>>> node > >>>>>>>> like > >>>>>>>>>> below did not work. > >>>>>>>>>> export OMP_NUM_THREADS=4 > >>>>>>>>>> mpiexec -bynode -np 4 ./my_program&& #PBS -l nodes=2:ppn=8 > >>>>>>>>>> > >>>>>>>>>> The error message is as follows: > >>>>>>>>>> [node08.cluster:11946] [[30666,0],3] ORTE_ERROR_LOG: A message is > >>>>>>>>>> attempting to be sent to a process whose contact infor > >>>>>>>>>> mation is unknown in file rml_oob_send.c at line 316 > >>>>>>>>>> [node08.cluster:11946] [[30666,0],3] unable to find address for > >>>>>>>>>> [[30666,0],1] > >>>>>>>>>> [node08.cluster:11946] [[30666,0],3] ORTE_ERROR_LOG: A message is > >>>>>>>>>> attempting to be sent to a process whose contact infor > >>>>>>>>>> mation is unknown in file base/grpcomm_base_rollup.c at line 123 > >>>>>>>>>> > >>>>>>>>>> Here is our openmpi configuration: > >>>>>>>>>> ./configure \ > >>>>>>>>>> --prefix=/home/mishima/opt/mpi/openmpi-1.7rc8-pgi12.9 \ > >>>>>>>>>> --with-tm \ > >>>>>>>>>> --with-verbs \ > >>>>>>>>>> --disable-ipv6 \ > >>>>>>>>>> CC=pgcc CFLAGS="-fast -tp k8-64e" \ > >>>>>>>>>> CXX=pgCC CXXFLAGS="-fast -tp k8-64e" \ > >>>>>>>>>> F77=pgfortran FFLAGS="-fast -tp k8-64e" \ > >>>>>>>>>> FC=pgfortran FCFLAGS="-fast -tp k8-64e" > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Tetsuya Mishima > >>>>>>>>>> > >>>>>>>>>>> On Mar 17, 2013, at 10:55 PM, Gustavo > >>>> Correa<g...@ldeo.columbia.edu> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> In your example, have you tried not to modify the node file, > >>>>>>>>>>>> launch two mpi processes with mpiexec, and request a "-bynode" > >>>>>>>>>> distribution of processes: > >>>>>>>>>>>> > >>>>>>>>>>>> mpiexec -bynode -np 2 ./my_program > >>>>>>>>>>> > >>>>>>>>>>> This should work in 1.7, too (I use these kinds of options with > >>>>>> SLURM > >>>>>>>> all > >>>>>>>>>> the time). > >>>>>>>>>>> > >>>>>>>>>>> However, we should probably verify that the hostfile > >> functionality > >>>>>> in > >>>>>>>>>> batch jobs hasn't been broken in 1.7, too, because I'm pretty > >> sure > >>>>>> that > >>>>>>>>>> what you described should work. However, Ralph, our > >>>>>>>>>>> run-time guy, is on vacation this week. There might be a delay > >> in > >>>>>>>>>> checking into this. > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Jeff Squyres > >>>>>>>>>>> jsquy...@cisco.com > >>>>>>>>>>> For corporate legal information go to: > >>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> users mailing list > >>>>>>>>>>> us...@open-mpi.org > >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> users mailing list > >>>>>>>>>> us...@open-mpi.org > >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Jeff Squyres > >>>>>>>>> jsquy...@cisco.com > >>>>>>>>> For corporate legal information go to: > >>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> users mailing list > >>>>>>>>> us...@open-mpi.org > >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> users mailing list > >>>>>>>> us...@open-mpi.org > >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> users mailing list > >>>>>>> us...@open-mpi.org > >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> us...@open-mpi.org > >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> > >>>> > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> > >>> > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
