You obviously have some MCA params set somewhere: > -------------------------------------------------------------------------- > A deprecated MCA parameter value was specified in an MCA parameter > file. Deprecated MCA parameters should be avoided; they may disappear > in future releases. > > Deprecated parameter: orte_rsh_agent > --------------------------------------------------------------------------
Check your environment for anything with OMPI_MCA_xxx, and your default MCA parameter file to see what has been specified. The allocation looks okay - I'll have to look for other debug flags you can set. Meantime, can you please add --enable-debug to your configure cmd line and rebuild? Thanks Ralph On Mar 20, 2013, at 4:39 PM, tmish...@jcity.maeda.co.jp wrote: > > > Hi Ralph, > > Here is a result of rerun with --display-allocation. > I set OMP_NUM_THREADS=1 to make the problem clear. > > Regards, > Tetsuya Mishima > > P.S. As far as I checked, these 2 cases are OK(no problem). > (1)mpirun -v -np $NPROCS-x OMP_NUM_THREADS --display-allocation > ~/Ducom/testbed/mPre m02-ld > (2)mpirun -v -x OMP_NUM_THREADS --display-allocation ~/Ducom/testbed/mPre > m02-ld > > Script File: > > #!/bin/sh > #PBS -A tmishima > #PBS -N Ducom-run > #PBS -j oe > #PBS -l nodes=2:ppn=4 > export OMP_NUM_THREADS=1 > cd $PBS_O_WORKDIR > cp $PBS_NODEFILE pbs_hosts > NPROCS=`wc -l < pbs_hosts` > mpirun -v -np $NPROCS -hostfile pbs_hosts -x OMP_NUM_THREADS > --display-allocation ~/Ducom/testbed/mPre m02-ld > > Output: > -------------------------------------------------------------------------- > A deprecated MCA parameter value was specified in an MCA parameter > file. Deprecated MCA parameters should be avoided; they may disappear > in future releases. > > Deprecated parameter: orte_rsh_agent > -------------------------------------------------------------------------- > > ====================== ALLOCATED NODES ====================== > > Data for node: node06 Num slots: 4 Max slots: 0 > Data for node: node05 Num slots: 4 Max slots: 0 > > ================================================================= > -------------------------------------------------------------------------- > A hostfile was provided that contains at least one node not > present in the allocation: > > hostfile: pbs_hosts > node: node06 > > If you are operating in a resource-managed environment, then only > nodes that are in the allocation can be used in the hostfile. You > may find relative node syntax to be a useful alternative to > specifying absolute node names see the orte_hosts man page for > further information. > -------------------------------------------------------------------------- > > >> I've submitted a patch to fix the Torque launch issue - just some > leftover garbage that existed at the time of the 1.7.0 branch and didn't > get removed. >> >> For the hostfile issue, I'm stumped as I can't see how the problem would > come about. Could you please rerun your original test and add > "--display-allocation" to your cmd line? Let's see if it is >> correctly finding the original allocation. >> >> Thanks >> Ralph >> >> On Mar 19, 2013, at 5:08 PM, tmish...@jcity.maeda.co.jp wrote: >> >>> >>> >>> Hi Gus, >>> >>> Thank you for your comments. I understand your advice. >>> Our script used to be --npernode type as well. >>> >>> As I told before, our cluster consists of nodes having 4, 8, >>> and 32 cores, although it used to be homogeneous at the >>> starting time. Furthermore, since performance of each core >>> is almost same, a mixed use of nodes with different number >>> of cores is possible, just like #PBS -l nodes=1:ppn=32+4:ppn=8. >>> >>> --npernode type is not applicable to such a mixed use. >>> That's why I'd like to continue to use modified hostfile. >>> >>> By the way, the problem I reported to Jeff yesterday >>> was that openmpi-1.7 with torque is something wrong, >>> because it caused error against such a simple case as >>> shown below, which surprised me. Now, the problem is not >>> limited to modified hostfile, I guess. >>> >>> #PBS -l nodes=4:ppn=8 >>> mpirun -np 8 ./my_program >>> (OMP_NUM_THREADS=4) >>> >>> Regards, >>> Tetsuya Mishima >>> >>>> Hi Tetsuya >>>> >>>> Your script that edits $PBS_NODEFILE into a separate hostfile >>>> is very similar to some that I used here for >>>> hybrid OpenMP+MPI programs on older versions of OMPI. >>>> I haven't tried this in 1.6.X, >>>> but it looks like you did and it works also. >>>> I haven't tried 1.7 either. >>>> Since we run production machines, >>>> I try to stick to the stable versions of OMPI (even numbered: >>>> 1.6.X, 1.4.X, 1.2.X). >>>> >>>> I believe you can get the same effect even if you >>>> don't edit your $PBS_NODEFILE and let OMPI use it as is. >>>> Say, if you choose carefully the values in your >>>> #PBS -l nodes=?:ppn=? >>>> of your >>>> $OMP_NUM_THREADS >>>> and use an mpiexec with --npernode or --cpus-per-proc. >>>> >>>> For instance, for twelve MPI processes, with two threads each, >>>> on nodes with eight cores each, I would try >>>> (but I haven't tried!): >>>> >>>> #PBS -l nodes=3:ppn=8 >>>> >>>> export $OMP_NUM_THREADS=2 >>>> >>>> mpiexec -np 12 -npernode 4 >>>> >>>> or perhaps more tightly: >>>> >>>> mpiexec -np 12 --report-bindings --bind-to-core --cpus-per-proc 2 >>>> >>>> I hope this helps, >>>> Gus Correa >>>> >>>> >>>> >>>> On 03/19/2013 03:12 PM, tmish...@jcity.maeda.co.jp wrote: >>>>> >>>>> >>>>> Hi Reuti and Gus, >>>>> >>>>> Thank you for your comments. >>>>> >>>>> Our cluster is a little bit heterogeneous, which has nodes with 4, 8, >>> 32 >>>>> cores. >>>>> I used 8-core nodes for "-l nodes=4:ppn=8" and 4-core nodes for "-l >>>>> nodes=2:ppn=4". >>>>> (strictly speaking, Torque picked up proper nodes.) >>>>> >>>>> As I mentioned before, I usually use openmpi-1.6.x, which has no > troble >>>>> against that kind >>>>> of use. I encountered the issue when I was evaluating openmpi-1.7 to >>> check >>>>> when we could >>>>> move on to it, although we have no positive reason to do that at this >>>>> moment. >>>>> >>>>> As Gus pointed out, I use a script file as shown below for a > practical >>> use >>>>> of openmpi-1.6.x. >>>>> >>>>> #PBS -l nodes=2:ppn=32 # even "-l nodes=1:ppn=32+4:ppn=8" works fine >>>>> export OMP_NUM_THREADS=4 >>>>> modify $PBS_NODEFILE pbs_hosts # 64 lines are condensed to 16 lines >>> here >>>>> mpirun -hostfile pbs_hosts -np 16 -cpus-per-proc 4 -report-bindings \ >>>>> -x OMP_NUM_THREADS ./my_program # 32-core node has 8 numanodes, > 8-core >>>>> node has 2 numanodes >>>>> >>>>> It works well under the combination of openmpi-1.6.x and Torque. The >>>>> problem is just >>>>> openmpi-1.7's behavior. >>>>> >>>>> Regards, >>>>> Tetsuya Mishima >>>>> >>>>>> Hi Tetsuya Mishima >>>>>> >>>>>> Mpiexec offers you a number of possibilities that you could try: >>>>>> --bynode, >>>>>> --pernode, >>>>>> --npernode, >>>>>> --bysocket, >>>>>> --bycore, >>>>>> --cpus-per-proc, >>>>>> --cpus-per-rank, >>>>>> --rankfile >>>>>> and more. >>>>>> >>>>>> Most likely one or more of them will fit your needs. >>>>>> >>>>>> There are also associated flags to bind processes to cores, >>>>>> to sockets, etc, to report the bindings, and so on. >>>>>> >>>>>> Check the mpiexec man page for details. >>>>>> >>>>>> Nevertheless, I am surprised that modifying the >>>>>> $PBS_NODEFILE doesn't work for you in OMPI 1.7. >>>>>> I have done this many times in older versions of OMPI. >>>>>> >>>>>> Would it work for you to go back to the stable OMPI 1.6.X, >>>>>> or does it lack any special feature that you need? >>>>>> >>>>>> I hope this helps, >>>>>> Gus Correa >>>>>> >>>>>> On 03/19/2013 03:00 AM, tmish...@jcity.maeda.co.jp wrote: >>>>>>> >>>>>>> >>>>>>> Hi Jeff, >>>>>>> >>>>>>> I didn't have much time to test this morning. So, I checked it > again >>>>>>> now. Then, the trouble seems to depend on the number of nodes to > use. >>>>>>> >>>>>>> This works(nodes< 4): >>>>>>> mpiexec -bynode -np 4 ./my_program&& #PBS -l nodes=2:ppn=8 >>>>>>> (OMP_NUM_THREADS=4) >>>>>>> >>>>>>> This causes error(nodes>= 4): >>>>>>> mpiexec -bynode -np 8 ./my_program&& #PBS -l nodes=4:ppn=8 >>>>>>> (OMP_NUM_THREADS=4) >>>>>>> >>>>>>> Regards, >>>>>>> Tetsuya Mishima >>>>>>> >>>>>>>> Oy; that's weird. >>>>>>>> >>>>>>>> I'm afraid we're going to have to wait for Ralph to answer why > that >>> is >>>>>>> happening -- sorry! >>>>>>>> >>>>>>>> >>>>>>>> On Mar 18, 2013, at 4:45 PM,<tmish...@jcity.maeda.co.jp> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Correa and Jeff, >>>>>>>>> >>>>>>>>> Thank you for your comments. I quickly checked your suggestion. >>>>>>>>> >>>>>>>>> As a result, my simple example case worked well. >>>>>>>>> export OMP_NUM_THREADS=4 >>>>>>>>> mpiexec -bynode -np 2 ./my_program&& #PBS -l nodes=2:ppn=4 >>>>>>>>> >>>>>>>>> But, practical case that more than 1 process was allocated to a >>> node >>>>>>> like >>>>>>>>> below did not work. >>>>>>>>> export OMP_NUM_THREADS=4 >>>>>>>>> mpiexec -bynode -np 4 ./my_program&& #PBS -l nodes=2:ppn=8 >>>>>>>>> >>>>>>>>> The error message is as follows: >>>>>>>>> [node08.cluster:11946] [[30666,0],3] ORTE_ERROR_LOG: A message is >>>>>>>>> attempting to be sent to a process whose contact infor >>>>>>>>> mation is unknown in file rml_oob_send.c at line 316 >>>>>>>>> [node08.cluster:11946] [[30666,0],3] unable to find address for >>>>>>>>> [[30666,0],1] >>>>>>>>> [node08.cluster:11946] [[30666,0],3] ORTE_ERROR_LOG: A message is >>>>>>>>> attempting to be sent to a process whose contact infor >>>>>>>>> mation is unknown in file base/grpcomm_base_rollup.c at line 123 >>>>>>>>> >>>>>>>>> Here is our openmpi configuration: >>>>>>>>> ./configure \ >>>>>>>>> --prefix=/home/mishima/opt/mpi/openmpi-1.7rc8-pgi12.9 \ >>>>>>>>> --with-tm \ >>>>>>>>> --with-verbs \ >>>>>>>>> --disable-ipv6 \ >>>>>>>>> CC=pgcc CFLAGS="-fast -tp k8-64e" \ >>>>>>>>> CXX=pgCC CXXFLAGS="-fast -tp k8-64e" \ >>>>>>>>> F77=pgfortran FFLAGS="-fast -tp k8-64e" \ >>>>>>>>> FC=pgfortran FCFLAGS="-fast -tp k8-64e" >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Tetsuya Mishima >>>>>>>>> >>>>>>>>>> On Mar 17, 2013, at 10:55 PM, Gustavo >>> Correa<g...@ldeo.columbia.edu> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> In your example, have you tried not to modify the node file, >>>>>>>>>>> launch two mpi processes with mpiexec, and request a "-bynode" >>>>>>>>> distribution of processes: >>>>>>>>>>> >>>>>>>>>>> mpiexec -bynode -np 2 ./my_program >>>>>>>>>> >>>>>>>>>> This should work in 1.7, too (I use these kinds of options with >>>>> SLURM >>>>>>> all >>>>>>>>> the time). >>>>>>>>>> >>>>>>>>>> However, we should probably verify that the hostfile > functionality >>>>> in >>>>>>>>> batch jobs hasn't been broken in 1.7, too, because I'm pretty > sure >>>>> that >>>>>>>>> what you described should work. However, Ralph, our >>>>>>>>>> run-time guy, is on vacation this week. There might be a delay > in >>>>>>>>> checking into this. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Jeff Squyres >>>>>>>>>> jsquy...@cisco.com >>>>>>>>>> For corporate legal information go to: >>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Squyres >>>>>>>> jsquy...@cisco.com >>>>>>>> For corporate legal information go to: >>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
