Guys
I changed the line to run the program in the script with both options
/usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v --bind-to-none -np $NSLOTS
./inverse.exe
/usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v --bind-to-socket -np $NSLOTS
./inverse.exe
but I got the same results. When I use man mpirun appears:
-bind-to-none, --bind-to-none
Do not bind processes. (Default.)
and the output of 'qconf -sp orte' is
pe_name orte
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule $fill_up
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE
I don't know if the installed Open MPI was compiled with '--with-sge'. How can
i know that?
before to think in an hybrid application i was using only MPI and the program
used few processors (14). The cluster possesses 28 machines, 15 with 16 cores
and 13 with 8 cores totalizing 344 units of processing. When I submitted the
job (only MPI), the MPI processes were spread to the cores directly, for that
reason I created a new queue with 14 machines trying to gain more time. the
results were the same in both cases. In the last case i could prove that the
processes were distributed to all machines correctly.
What I must to do?
Thanks
Oscar Fabian Mojica Ladino
Geologist M.S. in Geophysics
> Date: Thu, 14 Aug 2014 10:10:17 -0400
> From: [email protected]
> To: [email protected]
> Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
>
> Hi,
> You DEFINITELY need to disable OpenMPI's new default binding. Otherwise,
> your N threads will run on a single core. --bind-to socket would be my
> recommendation for hybrid jobs.
>
> Maxime
>
>
> Le 2014-08-14 10:04, Jeff Squyres (jsquyres) a écrit :
> > I don't know much about OpenMP, but do you need to disable Open MPI's
> > default bind-to-core functionality (I'm assuming you're using Open MPI
> > 1.8.x)?
> >
> > You can try "mpirun --bind-to none ...", which will have Open MPI not bind
> > MPI processes to cores, which might allow OpenMP to think that it can use
> > all the cores, and therefore it will spawn num_cores threads...?
> >
> >
> > On Aug 14, 2014, at 9:50 AM, Oscar Mojica <[email protected]> wrote:
> >
> >> Hello everybody
> >>
> >> I am trying to run a hybrid mpi + openmp program in a cluster. I created
> >> a queue with 14 machines, each one with 16 cores. The program divides the
> >> work among the 14 processors with MPI and within each processor a loop is
> >> also divided into 8 threads for example, using openmp. The problem is that
> >> when I submit the job to the queue the MPI processes don't divide the work
> >> into threads and the program prints the number of threads that are
> >> working within each process as one.
> >>
> >> I made a simple test program that uses openmp and I logged in one machine
> >> of the fourteen. I compiled it using gfortran -fopenmp program.f -o exe,
> >> set the OMP_NUM_THREADS environment variable equal to 8 and when I ran
> >> directly in the terminal the loop was effectively divided among the cores
> >> and for example in this case the program printed the number of threads
> >> equal to 8
> >>
> >> This is my Makefile
> >>
> >> # Start of the makefile
> >> # Defining variables
> >> objects = inv_grav3d.o funcpdf.o gr3dprm.o fdjac.o dsvd.o
> >> #f90comp = /opt/openmpi/bin/mpif90
> >> f90comp = /usr/bin/mpif90
> >> #switch = -O3
> >> executable = inverse.exe
> >> # Makefile
> >> all : $(executable)
> >> $(executable) : $(objects)
> >> $(f90comp) -fopenmp -g -O -o $(executable) $(objects)
> >> rm $(objects)
> >> %.o: %.f
> >> $(f90comp) -c $<
> >> # Cleaning everything
> >> clean:
> >> rm $(executable)
> >> # rm $(objects)
> >> # End of the makefile
> >>
> >> and the script that i am using is
> >>
> >> #!/bin/bash
> >> #$ -cwd
> >> #$ -j y
> >> #$ -S /bin/bash
> >> #$ -pe orte 14
> >> #$ -N job
> >> #$ -q new.q
> >>
> >> export OMP_NUM_THREADS=8
> >> /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v -np $NSLOTS ./inverse.exe
> >>
> >> am I forgetting something?
> >>
> >> Thanks,
> >>
> >> Oscar Fabian Mojica Ladino
> >> Geologist M.S. in Geophysics
> >> _______________________________________________
> >> users mailing list
> >> [email protected]
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> Link to this post:
> >> http://www.open-mpi.org/community/lists/users/2014/08/25016.php
> >
>
>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Québec, Université Laval
> Ph. D. en physique
>
> _______________________________________________
> users mailing list
> [email protected]
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/08/25020.php
