Hi John, We have Univa grid 8.6.7
I will get information about the network from our IT and respond back. The grid has 1000+ nodes. Regards, Vipul From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of John Hearns via users Sent: Saturday, May 30, 2020 2:19 AM To: Open MPI Users <users@lists.open-mpi.org> Cc: John Hearns <hear...@gmail.com> Subject: Re: [OMPI users] Running mpirun with grid Good morning Vipul. I would like to ask some higher level questions regarding your HPC cluster. What are the manufacturers of the cluster nodes. How many compute nodes? What network interconnect do you have - gigabit ethernet, 10gig ethernet, Infiniband, Omnipath? Which cluster middleware - openHPC? Rocks? Bright? Qlustar? Which version of grid - there have been MANY versions of this over the years. Who installed the cluster ? And now the big question - and everyone on the list will laugh at me for this.... Would you consider switching to using the Slurm batch queuing system? On Sat, 30 May 2020 at 00:41, Kulshrestha, Vipul via users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>> wrote: Hi, I need to launch my openmpi application on grid. My application is designed to run N processes, where each process would have M threads. I am using open MPI version 4.0.1 % /build/openmpi/openmpi-4.0.1/rhel6/bin/ompi_info | grep grid MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component v4.0.1) To run it without grid, I run it as (say N = 7, M = 2) % mpirun –np 7 <application with arguments> The above works well and runs N processes. Based on some earlier advice on this forum, I have setup the grid submission using the a grid job submission script that modifies the grid slow allocation, so that mpirun launches only 1 application process copy on each host allocated by grid. I have some partial success. I think grid is able to start the job and then mpirun also starts to run, but then it errors out with below mentioned errors. Strangely, after giving message for having started all the daemons, it reports that it was not able to start one or more daemons. I have setup a grid submission script that modifies the pe_hostfile and it appears that mpirun is able to take it and then is able use the host information to start launching the jobs. However, mpirun halts before it can start all the child processes. I enabled some debug logs but am not able to figure out a possible cause. Could somebody look at this and guide how to resolve this issue? I have pasted the detailed log as well as my job submission script below. As a clarification, when I run the mpirun without grid, it (mpirun and my application) works on the same set of hosts without any problems. Thanks, Vipul Job submission script: #!/bin/sh #$ -N velsyn #$ -pe orte2 14 #$ -V -cwd -j y #$ -o out.txt # echo "Got $NSLOTS slots." echo "tmpdir is $TMPDIR" echo "pe_hostfile is $PE_HOSTFILE" cat $PE_HOSTFILE newhostfile=/testdir/tmp/pe_hostfile awk '{$2 = $2/2; print}' $PE_HOSTFILE > $newhostfile export PE_HOSTFILE=$newhostfile export LD_LIBRARY_PATH=/build/openmpi/openmpi-4.0.1/rhel6/lib mpirun --merge-stderr-to-stdout --output-filename ./output:nojobid,nocopy --mca routed direct --mca orte_base_help_aggregate 0 --mca plm_base_verbose 1 --bind-to none --report-bindings -np 7 <application with args> The out.txt content is: Got 14 slots. tmpdir is /tmp/182117160.1.all.q pe_hostfile is /var/spool/sge/bos2/active_jobs/182117160.1/pe_hostfile bos2.wv.org.com<http://bos2.wv.org.com> 2 al...@bos2.wv.org.com<mailto:al...@bos2.wv.org.com> <NULL> art8.wv.org.com<http://art8.wv.org.com> 2 al...@art8.wv.org.com<mailto:al...@art8.wv.org.com> <NULL> art10.wv.org.com<http://art10.wv.org.com> 2 al...@art10.wv.org.com<mailto:al...@art10.wv.org.com> <NULL> hpb7.wv.org.com<http://hpb7.wv.org.com> 2 al...@hpb7.wv.org.com<mailto:al...@hpb7.wv.org.com> <NULL> bos15.wv.org.com<http://bos15.wv.org.com> 2 al...@bos15.wv.org.com<mailto:al...@bos15.wv.org.com> <NULL> bos1.wv.org.com<http://bos1.wv.org.com> 2 al...@bos1.wv.org.com<mailto:al...@bos1.wv.org.com> <NULL> hpb11.wv.org.com<http://hpb11.wv.org.com> 2 al...@hpb11.wv.org.com<mailto:al...@hpb11.wv.org.com> <NULL> [bos2:22657] [[8251,0],0] plm:rsh: using "/wv/grid2/sge/bin/lx-amd64/qrsh -inherit -nostdin -V -verbose" for launching [bos2:22657] [[8251,0],0] plm:rsh: final template argv: /grid2/sge/bin/lx-amd64/qrsh -inherit -nostdin -V -verbose <template> set path = ( /build/openm pi/openmpi-4.0.1/rhel6/bin $path ) ; if ( $?LD_LIBRARY_PATH == 1 ) set OMPI_have_llp ; if ( $?LD_LIBR ARY_PATH == 0 ) setenv LD_LIBRARY_PATH /build/openmpi/openmpi-4.0.1/rhel6/lib ; if ( $?OMPI_have_llp == 1 ) setenv LD_LIBRARY_PATH /build/openmpi/openmpi-4.0.1/rhel6/lib:$LD_LIBRARY_PATH ; if ( $?DYLD_L IBRARY_PATH == 1 ) set OMPI_have_dllp ; if ( $?DYLD_LIBRARY_PATH == 0 ) setenv DYLD_LIBRARY_PATH /bui ld/openmpi/openmpi-4.0.1/rhel6/lib ; if ( $?OMPI_have_dllp == 1 ) setenv DYLD_LIBRARY_PATH /build/ope nmpi/openmpi-4.0.1/rhel6/lib:$DYLD_LIBRARY_PATH ; /build/openmpi/openmpi-4.0.1/rhel6/bin/orted -mca orte_report_bindings "1" -mca ess "env" -mca ess_base_jobid "540737536" -mca ess_base_vpid "<templat e>" -mca ess_base_num_procs "7" -mca orte_node_regex e>"bos[1:2],art[1:8],art[2:10],hpb[1:7],bos[2:15], bos[1:1],hpb[2:11]@0(7)" -mca orte_hnp_uri "540737536.0;tcp://147.34.116.60:50769<http://147.34.116.60:50769>" --mca routed "dire ct" --mca orte_base_help_aggregate "0" --mca plm_base_verbose "1" -mca plm "rsh" --tree-spawn -mca or te_parent_uri "540737536.0;tcp://147.34.116.60:50769<http://147.34.116.60:50769>" -mca orte_output_filename "./output:nojobid,noc opy" -mca hwloc_base_binding_policy "none" -mca hwloc_base_report_bindings "1" -mca pmix "^s1,s2,cray ,isolated" Starting server daemon at host "art10" Starting server daemon at host "art8" Starting server daemon at host "bos1" Starting server daemon at host "hpb7" Starting server daemon at host "hpb11" Starting server daemon at host "bos15" Server daemon successfully started with task id "1.art8" Server daemon successfully started with task id "1.bos1" Server daemon successfully started with task id "1.art10" Server daemon successfully started with task id "1.bos15" Server daemon successfully started with task id "1.hpb7" Server daemon successfully started with task id "1.hpb11" Unmatched ". -------------------------------------------------------------------------- ORTE was unable to reliably start one or more daemons. This usually is caused by: * not finding the required libraries and/or binaries on one or more nodes. Please check your PATH and LD_LIBRARY_PATH settings, or configure OMPI with --enable-orterun-prefix-by-default * lack of authority to execute on one or more specified nodes. Please verify your allocation and authorities. * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base). Please check with your sys admin to determine the correct location to use. * compilation of the orted with dynamic libraries when static are required (e.g., on Cray). Please check your configure cmd line and consider using one of the contrib/platform definitions for your system type. * an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements). -------------------------------------------------------------------------- -------------------------------------------------------------------------- ORTE does not know how to route a message to the specified daemon located on the indicated node: my node: bos2 target node: art10 This is usually an internal programming error that should be reported to the developers. In the meantime, a workaround may be to set the MCA param routed=direct on the command line or in your environment. We apologize for the problem. -------------------------------------------------------------------------- -------------------------------------------------------------------------- ORTE does not know how to route a message to the specified daemon located on the indicated node: my node: bos2 target node: hpb7 This is usually an internal programming error that should be reported to the developers. In the meantime, a workaround may be to set the MCA param routed=direct on the command line or in your environment. We apologize for the problem. -------------------------------------------------------------------------- -------------------------------------------------------------------------- ORTE does not know how to route a message to the specified daemon located on the indicated node: my node: bos2 target node: bos15 This is usually an internal programming error that should be reported to the developers. In the meantime, a workaround may be to set the MCA param routed=direct on the command line or in your environment. We apologize for the problem. -------------------------------------------------------------------------- -------------------------------------------------------------------------- ORTE does not know how to route a message to the specified daemon located on the indicated node: my node: bos2 target node: bos1 This is usually an internal programming error that should be reported to the developers. In the meantime, a workaround may be to set the MCA param routed=direct on the command line or in your environment. We apologize for the problem. -------------------------------------------------------------------------- -------------------------------------------------------------------------- ORTE does not know how to route a message to the specified daemon located on the indicated node: my node: bos2 target node: hpb11 This is usually an internal programming error that should be reported to the developers. In the meantime, a workaround may be to set the MCA param routed=direct on the command line or in your environment. We apologize for the problem. --------------------------------------------------------------------------
