Reuti and Oscar,
I'm a Torque user and I myself have never used SGE, so I hesitated to join
the discussion.
>From my experience with the Torque, the openmpi 1.8 series has already
resolved the issue you pointed out in combining MPI with OpenMP.
Please try to add --map-by slot:pe=8 option, if you want to use 8 threads.
Then, then openmpi 1.8 should allocate processes properly without any
modification
of the hostfile provided by the Torque.
In your case(8 threads and 10 procs):
# you have to request 80 slots using SGE command before mpirun
mpirun --map-by slot:pe=8 -np 10 ./inverse.exe
where you can omit --bind-to option because --bind-to core is assumed
as default when pe=N is provided by the user.
Regards,
Tetsuya
>Hi,
>
>Am 19.08.2014 um 19:06 schrieb Oscar Mojica:
>
>> I discovered what was the error. I forgot include the '-fopenmp' when I
>> compiled the objects in the Makefile, so the program worked but it didn't
>> divide the job
in threads. Now the program is working and I can use until 15 cores for machine
in the queue one.q.
>>
>> Anyway i would like to try implement your advice. Well I'm not alone in the
>> cluster so i must implement your second suggestion. The steps are
>>
>> a) Use '$ qconf -mp orte' to change the allocation rule to 8
>
>The number of slots defined in your used one.q was also increased to 8 (`qconf
>-sq one.q`)?
>
>
>> b) Set '#$ -pe orte 80' in the script
>
>Fine.
>
>
>> c) I'm not sure how to do this step. I'd appreciate your help here. I can
>> add some lines to the script to determine the PE_HOSTFILE path and contents,
>> but i
don't know how alter it
>
>For now you can put in your jobscript (just after OMP_NUM_THREAD is exported):
>
>awk -v omp_num_threads=$OMP_NUM_THREADS '{ $2/=omp_num_threads; print }'
>$PE_HOSTFILE > $TMPDIR/machines
>export PE_HOSTFILE=$TMPDIR/machines
>
>=============
>
>Unfortunately noone stepped into this discussion, as in my opinion it's a much
>broader issue which targets all users who want to combine MPI with OpenMP. The
queuingsystem should get a proper request for the overall amount of slots the
user needs. For now this will be forwarded to Open MPI and it will use this
information to start the appropriate number of processes (which was an
achievement for the Tight Integration out-of-the-box of course) and ignores any
setting of
OMP_NUM_THREADS. So, where should the generated list of machines be adjusted;
there are several options:
>
>a) The PE of the queuingsystem should do it:
>
>+ a one time setup for the admin
>+ in SGE the "start_proc_args" of the PE could alter the $PE_HOSTFILE
>- the "start_proc_args" would need to know the number of threads, i.e.
>OMP_NUM_THREADS must be defined by "qsub -v ..." outside of the jobscript
>(tricky scanning
of the submitted jobscript for OMP_NUM_THREADS would be too nasty)
>- limits to use inside the jobscript calls to libraries behaving in the same
>way as Open MPI only
>
>
>b) The particular queue should do it in a queue prolog:
>
>same as a) I think
>
>
>c) The user should do it
>
>+ no change in the SGE installation
>- each and every user must include it in all the jobscripts to adjust the list
>and export the pointer to the $PE_HOSTFILE, but he could change it forth and
>back
for different steps of the jobscript though
>
>
>d) Open MPI should do it
>
>+ no change in the SGE installation
>+ no change to the jobscript
>+ OMP_NUM_THREADS can be altered for different steps of the jobscript while
>staying inside the granted allocation automatically
>o should MKL_NUM_THREADS be covered too (does it use OMP_NUM_THREADS already)?
>
>-- Reuti
>
>
>> echo "PE_HOSTFILE:"
>> echo $PE_HOSTFILE
>> echo
>> echo "cat PE_HOSTFILE:"
>> cat $PE_HOSTFILE
>>
>> Thanks for take a time for answer this emails, your advices had been very
>> useful
>>
>> PS: The version of SGE is OGS/GE 2011.11p1
>>
>>
>> Oscar Fabian Mojica Ladino
>> Geologist M.S. in Geophysics
>>
>>
>> > From: re...@staff.uni-marburg.de
>> > Date: Fri, 15 Aug 2014 20:38:12 +0200
>> > To: us...@open-mpi.org
>> > Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
>> >
>> > Hi,
>> >
>> > Am 15.08.2014 um 19:56 schrieb Oscar Mojica:
>> >
>> > > Yes, my installation of Open MPI is SGE-aware. I got the following
>> > >
>> > > [oscar@compute-1-2 ~]$ ompi_info | grep grid
>> > > MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.2)
>> >
>> > Fine.
>> >
>> >
>> > > I'm a bit slow and I didn't understand the las part of your message. So
>> > > i made a test trying to solve my doubts.
>> > > This is the cluster configuration: There are some machines turned off
>> > > but that is no problem
>> > >
>> > > [oscar@aguia free-noise]$ qhost
>> > > HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
>> > > -------------------------------------------------------------------------------
>> > > global - - - - - - -
>> > > compute-1-10 linux-x64 16 0.97 23.6G 558.6M 996.2M 0.0
>> > > compute-1-11 linux-x64 16 - 23.6G - 996.2M -
>> > > compute-1-12 linux-x64 16 0.97 23.6G 561.1M 996.2M 0.0
>> > > compute-1-13 linux-x64 16 0.99 23.6G 558.7M 996.2M 0.0
>> > > compute-1-14 linux-x64 16 1.00 23.6G 555.1M 996.2M 0.0
>> > > compute-1-15 linux-x64 16 0.97 23.6G 555.5M 996.2M 0.0
>> > > compute-1-16 linux-x64 8 0.00 15.7G 296.9M 1000.0M 0.0
>> > > compute-1-17 linux-x64 8 0.00 15.7G 299.4M 1000.0M 0.0
>> > > compute-1-18 linux-x64 8 - 15.7G - 1000.0M -
>> > > compute-1-19 linux-x64 8 - 15.7G - 996.2M -
>> > > compute-1-2 linux-x64 16 1.19 23.6G 468.1M 1000.0M 0.0
>> > > compute-1-20 linux-x64 8 0.04 15.7G 297.2M 1000.0M 0.0
>> > > compute-1-21 linux-x64 8 - 15.7G - 1000.0M -
>> > > compute-1-22 linux-x64 8 0.00 15.7G 297.2M 1000.0M 0.0
>> > > compute-1-23 linux-x64 8 0.16 15.7G 299.6M 1000.0M 0.0
>> > > compute-1-24 linux-x64 8 0.00 15.7G 291.5M 996.2M 0.0
>> > > compute-1-25 linux-x64 8 0.04 15.7G 293.4M 996.2M 0.0
>> > > compute-1-26 linux-x64 8 - 15.7G - 1000.0M -
>> > > compute-1-27 linux-x64 8 0.00 15.7G 297.0M 1000.0M 0.0
>> > > compute-1-29 linux-x64 8 - 15.7G - 1000.0M -
>> > > compute-1-3 linux-x64 16 - 23.6G - 996.2M -
>> > > compute-1-30 linux-x64 16 - 23.6G - 996.2M -
>> > > compute-1-4 linux-x64 16 0.97 23.6G 571.6M 996.2M 0.0
>> > > compute-1-5 linux-x64 16 1.00 23.6G 559.6M 996.2M 0.0
>> > > compute-1-6 linux-x64 16 0.66 23.6G 403.1M 996.2M 0.0
>> > > compute-1-7 linux-x64 16 0.95 23.6G 402.7M 996.2M 0.0
>> > > compute-1-8 linux-x64 16 0.97 23.6G 556.8M 996.2M 0.0
>> > > compute-1-9 linux-x64 16 1.02 23.6G 566.0M 1000.0M 0.0
>> > >
>> > > I ran my program using only MPI with 10 processors of the queue one.q
>> > > which has 14 machines (compute-1-2 to compute-1-15). Whit 'qstat -t' I
>> > > got:
>> > >
>> > > [oscar@aguia free-noise]$ qstat -t
>> > > job-ID prior name user state submit/start at queue master ja-task-ID
>> > > task-ID state cpu mem io stat failed
>> > >
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
----
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-2.local
>> > > MASTER r 00:49:12 554.13753 0.09163
>> > > one.q@compute-1-2.local SLAVE
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-5.local
>> > > SLAVE 1.compute-1-5 r 00:48:53 551.49022 0.09410
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-9.local
>> > > SLAVE 1.compute-1-9 r 00:50:00 564.22764 0.09409
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-12.local
>> > > SLAVE 1.compute-1-12 r 00:47:30 535.30379 0.09379
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-13.local
>> > > SLAVE 1.compute-1-13 r 00:49:51 561.69868 0.09379
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-14.local
>> > > SLAVE 1.compute-1-14 r 00:49:14 554.60818 0.09379
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-10.local
>> > > SLAVE 1.compute-1-10 r 00:49:59 562.95487 0.09349
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-15.local
>> > > SLAVE 1.compute-1-15 r 00:50:01 563.27221 0.09361
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-8.local
>> > > SLAVE 1.compute-1-8 r 00:49:26 556.68431 0.09349
>> > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-4.local
>> > > SLAVE 1.compute-1-4 r 00:49:27 556.87510 0.04967
>> >
>> > Yes, here you got 10 slots (= cores) granted by SGE. So there is no free
>> > core left inside the allocation of SGE to allow the use of additional
>> > cores for your
threads. If you use more cores than granted by SGE, it will oversubscribe the
machines.
>> >
>> > The issue is now:
>> >
>> > a) If you want 8 threads per MPI process, your job will use 80 cores in
>> > total - for now SGE isn't aware of it.
>> >
>> > b) Although you specified $fill_up as allocation rule, it looks like
>> > $round_robin. Is there more than one slot defined in the queue definition
>> > of one.q to get
exclusive access?
>> >
>> > c) What version of SGE are you using? Certain ones use cgroups or bind
>> > processes directly to cores (although it usually needs to be requested by
>> > the job:
first line of `qconf -help`).
>> >
>> >
>> > In case you are alone in the cluster, you could bypass the allocation with
>> > b) (unless you are hit by c)). But having a mixture of users and jobs a
>> > different
handling would be necessary to handle this in a proper way IMO:
>> >
>> > a) having a PE with a fixed allocation rule of 8
>> >
>> > b) requesting this PE with an overall slot count of 80
>> >
>> > c) copy and alter the $PE_HOSTFILE to show only (granted core count per
>> > machine) divided by (OMP_NUM_THREADS) per entry, change $PE_HOSTFILE so
>> > that it points
to the altered file
>> >
>> > d) Open MPI with a Tight Integration will now start only N process per
>> > machine according to the altered hostfile, in your case one
>> >
>> > e) Your application can start the desired threads and you stay inside the
>> > granted allocation
>> >
>> > -- Reuti
>> >
>> >
>> > > I accessed to the MASTER processor with 'ssh compute-1-2.local' , and
>> > > with $ ps -e f and got this, I'm showing only the last lines
>> > >
>> > > 2506 ? Ss 0:00 /usr/sbin/atd
>> > > 2548 tty1 Ss+ 0:00 /sbin/mingetty /dev/tty1
>> > > 2550 tty2 Ss+ 0:00 /sbin/mingetty /dev/tty2
>> > > 2552 tty3 Ss+ 0:00 /sbin/mingetty /dev/tty3
>> > > 2554 tty4 Ss+ 0:00 /sbin/mingetty /dev/tty4
>> > > 2556 tty5 Ss+ 0:00 /sbin/mingetty /dev/tty5
>> > > 2558 tty6 Ss+ 0:00 /sbin/mingetty /dev/tty6
>> > > 3325 ? Sl 0:04 /opt/gridengine/bin/linux-x64/sge_execd
>> > > 17688 ? S 0:00 \_ sge_shepherd-2726 -bg
>> > > 17695 ? Ss 0:00 \_ -bash
>> > > /opt/gridengine/default/spool/compute-1-2/job_scripts/2726
>> > > 17797 ? S 0:00 \_ /usr/bin/time -f %E /opt/openmpi/bin/mpirun -v -np 10
>> > > ./inverse.exe
>> > > 17798 ? S 0:01 \_ /opt/openmpi/bin/mpirun -v -np 10 ./inverse.exe
>> > > 17799 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-5.local PATH=/opt/openmpi/bin:$PATH ; expo
>> > > 17800 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-9.local PATH=/opt/openmpi/bin:$PATH ; expo
>> > > 17801 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-12.local PATH=/opt/openmpi/bin:$PATH ; exp
>> > > 17802 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-13.local PATH=/opt/openmpi/bin:$PATH ; exp
>> > > 17803 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-14.local PATH=/opt/openmpi/bin:$PATH ; exp
>> > > 17804 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-10.local PATH=/opt/openmpi/bin:$PATH ; exp
>> > > 17805 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-15.local PATH=/opt/openmpi/bin:$PATH ; exp
>> > > 17806 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-8.local PATH=/opt/openmpi/bin:$PATH ; expo
>> > > 17807 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
>> > > -V compute-1-4.local PATH=/opt/openmpi/bin:$PATH ; expo
>> > > 17826 ? R 31:36 \_ ./inverse.exe
>> > > 3429 ? Ssl 0:00 automount --pid-file /var/run/autofs.pid
>> > >
>> > > So the job is using the 10 machines, Until here is all right OK. Do you
>> > > think that changing the "allocation_rule " to a number instead $fill_up
>> > > the MPI
processes would divide the work in that number of threads?
>> > >
>> > > Thanks a lot
>> > >
>> > > Oscar Fabian Mojica Ladino
>> > > Geologist M.S. in Geophysics
>> > >
>> > >
>> > > PS: I have another doubt, what is a slot? is a physical core?
>> > >
>> > >
>> > > > From: re...@staff.uni-marburg.de
>> > > > Date: Thu, 14 Aug 2014 23:54:22 +0200
>> > > > To: us...@open-mpi.org
>> > > > Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
>> > > >
>> > > > Hi,
>> > > >
>> > > > I think this is a broader issue in case an MPI library is used in
>> > > > conjunction with threads while running inside a queuing system. First:
>> > > > whether your
actual installation of Open MPI is SGE-aware you can check with:
>> > > >
>> > > > $ ompi_info | grep grid
>> > > > MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.5)
>> > > >
>> > > > Then we can look at the definition of your PE: "allocation_rule
>> > > > $fill_up". This means that SGE will grant you 14 slots in total in any
>> > > > combination on the
available machines, means 8+4+2 slots allocation is an allowed combination like
4+4+3+3 and so on. Depending on the SGE-awareness it's a question: will your
application just start processes on all nodes and completely disregard the
granted allocation, or as the other extreme does it stays on one and the same
machine
for all started processes? On the master node of the parallel job you can issue:
>> > > >
>> > > > $ ps -e f
>> > > >
>> > > > (f w/o -) to have a look whether `ssh` or `qrsh -inhert ...` is used
>> > > > to reach other machines and their requested process count.
>> > > >
>> > > >
>> > > > Now to the common problem in such a set up:
>> > > >
>> > > > AFAICS: for now there is no way in the Open MPI + SGE combination to
>> > > > specify the number of MPI processes and intended number of threads
>> > > > which are
automatically read by Open MPI while staying inside the granted slot count and
allocation. So it seems to be necessary to have the intended number of threads
being
honored by Open MPI too.
>> > > >
>> > > > Hence specifying e.g. "allocation_rule 8" in such a setup while
>> > > > requesting 32 processes, would for now start 32 processes by MPI
>> > > > already, as Open MP reads
the $PE_HOSTFILE and acts accordingly.
>> > > >
>> > > > Open MPI would have to read the generated machine file in a slightly
>> > > > different way regarding threads: a) read the $PE_HOSTFILE, b) divide
>> > > > the granted
slots per machine by OMP_NUM_THREADS, c) throw an error in case it's not
divisible by OMP_NUM_THREADS. Then start one process per quotient.
>> > > >
>> > > > Would this work for you?
>> > > >
>> > > > -- Reuti
>> > > >
>> > > > PS: This would also mean to have a couple of PEs in SGE having a fixed
>> > > > "allocation_rule". While this works right now, an extension in SGE
>> > > > could be
"$fill_up_omp"/"$round_robin_omp" and using OMP_NUM_THREADS there too, hence it
must not be specified as an `export` in the job script but either on the
command
line or inside the job script in #$ lines as job requests. This would mean to
collect slots in bunches of OMP_NUM_THREADS on each machine to reach the
overall
specified slot count. Whether OMP_NUM_THREADS or n times OMP_NUM_THREADS is
allowed per machine needs to be discussed.
>> > > >
>> > > > PS2: As Univa SGE can also supply a list of granted cores in the
>> > > > $PE_HOSTFILE, it would be an extension to feed this to Open MPI to
>> > > > allow any UGE aware
binding.
>> > > >
>> > > >
>> > > > Am 14.08.2014 um 21:52 schrieb Oscar Mojica:
>> > > >
>> > > > > Guys
>> > > > >
>> > > > > I changed the line to run the program in the script with both options
>> > > > > /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v --bind-to-none -np
>> > > > > $NSLOTS ./inverse.exe
>> > > > > /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v --bind-to-socket
>> > > > > -np $NSLOTS ./inverse.exe
>> > > > >
>> > > > > but I got the same results. When I use man mpirun appears:
>> > > > >
>> > > > > -bind-to-none, --bind-to-none
>> > > > > Do not bind processes. (Default.)
>> > > > >
>> > > > > and the output of 'qconf -sp orte' is
>> > > > >
>> > > > > pe_name orte
>> > > > > slots 9999
>> > > > > user_lists NONE
>> > > > > xuser_lists NONE
>> > > > > start_proc_args /bin/true
>> > > > > stop_proc_args /bin/true
>> > > > > allocation_rule $fill_up
>> > > > > control_slaves TRUE
>> > > > > job_is_first_task FALSE
>> > > > > urgency_slots min
>> > > > > accounting_summary TRUE
>> > > > >
>> > > > > I don't know if the installed Open MPI was compiled with
>> > > > > '--with-sge'. How can i know that?
>> > > > > before to think in an hybrid application i was using only MPI and
>> > > > > the program used few processors (14). The cluster possesses 28
>> > > > > machines, 15 with 16
cores and 13 with 8 cores totalizing 344 units of processing. When I submitted
the job (only MPI), the MPI processes were spread to the cores directly, for
that
reason I created a new queue with 14 machines trying to gain more time. the
results were the same in both cases. In the last case i could prove that the
processes
were distributed to all machines correctly.
>> > > > >
>> > > > > What I must to do?
>> > > > > Thanks
>> > > > >
>> > > > > Oscar Fabian Mojica Ladino
>> > > > > Geologist M.S. in Geophysics
>> > > > >
>> > > > >
>> > > > > > Date: Thu, 14 Aug 2014 10:10:17 -0400
>> > > > > > From: maxime.boissonnea...@calculquebec.ca
>> > > > > > To: us...@open-mpi.org
>> > > > > > Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
>> > > > > >
>> > > > > > Hi,
>> > > > > > You DEFINITELY need to disable OpenMPI's new default binding.
>> > > > > > Otherwise,
>> > > > > > your N threads will run on a single core. --bind-to socket would
>> > > > > > be my
>> > > > > > recommendation for hybrid jobs.
>> > > > > >
>> > > > > > Maxime
>> > > > > >
>> > > > > >
>> > > > > > Le 2014-08-14 10:04, Jeff Squyres (jsquyres) a 馗rit :
>> > > > > > > I don't know much about OpenMP, but do you need to disable Open
>> > > > > > > MPI's default bind-to-core functionality (I'm assuming you're
>> > > > > > > using Open MPI 1.8.x)?
>> > > > > > >
>> > > > > > > You can try "mpirun --bind-to none ...", which will have Open
>> > > > > > > MPI not bind MPI processes to cores, which might allow OpenMP to
>> > > > > > > think that it can use
all the cores, and therefore it will spawn num_cores threads...?
>> > > > > > >
>> > > > > > >
>> > > > > > > On Aug 14, 2014, at 9:50 AM, Oscar Mojica
>> > > > > > > <o_moji...@hotmail.com> wrote:
>> > > > > > >
>> > > > > > >> Hello everybody
>> > > > > > >>
>> > > > > > >> I am trying to run a hybrid mpi + openmp program in a cluster.
>> > > > > > >> I created a queue with 14 machines, each one with 16 cores. The
>> > > > > > >> program divides the
work among the 14 processors with MPI and within each processor a loop is also
divided into 8 threads for example, using openmp. The problem is that when I
submit
the job to the queue the MPI processes don't divide the work into threads and
the program prints the number of threads that are working within each process
as one.
>> > > > > > >>
>> > > > > > >> I made a simple test program that uses openmp and I logged in
>> > > > > > >> one machine of the fourteen. I compiled it using gfortran
>> > > > > > >> -fopenmp program.f -o exe,
set the OMP_NUM_THREADS environment variable equal to 8 and when I ran directly
in the terminal the loop was effectively divided among the cores and for
example in
this case the program printed the number of threads equal to 8
>> > > > > > >>
>> > > > > > >> This is my Makefile
>> > > > > > >>
>> > > > > > >> # Start of the makefile
>> > > > > > >> # Defining variables
>> > > > > > >> objects = inv_grav3d.o funcpdf.o gr3dprm.o fdjac.o dsvd.o
>> > > > > > >> #f90comp = /opt/openmpi/bin/mpif90
>> > > > > > >> f90comp = /usr/bin/mpif90
>> > > > > > >> #switch = -O3
>> > > > > > >> executable = inverse.exe
>> > > > > > >> # Makefile
>> > > > > > >> all : $(executable)
>> > > > > > >> $(executable) : $(objects)
>> > > > > > >> $(f90comp) -fopenmp -g -O -o $(executable) $(objects)
>> > > > > > >> rm $(objects)
>> > > > > > >> %.o: %.f
>> > > > > > >> $(f90comp) -c $<
>> > > > > > >> # Cleaning everything
>> > > > > > >> clean:
>> > > > > > >> rm $(executable)
>> > > > > > >> # rm $(objects)
>> > > > > > >> # End of the makefile
>> > > > > > >>
>> > > > > > >> and the script that i am using is
>> > > > > > >>
>> > > > > > >> #!/bin/bash
>> > > > > > >> #$ -cwd
>> > > > > > >> #$ -j y
>> > > > > > >> #$ -S /bin/bash
>> > > > > > >> #$ -pe orte 14
>> > > > > > >> #$ -N job
>> > > > > > >> #$ -q new.q
>> > > > > > >>
>> > > > > > >> export OMP_NUM_THREADS=8
>> > > > > > >> /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v -np $NSLOTS
>> > > > > > >> ./inverse.exe
>> > > > > > >>
>> > > > > > >> am I forgetting something?
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >>
>> > > > > > >> Oscar Fabian Mojica Ladino
>> > > > > > >> Geologist M.S. in Geophysics
>> > > > > > >> _______________________________________________
>> > > > > > >> users mailing list
>> > > > > > >> us...@open-mpi.org
>> > > > > > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > > > > > >> Link to this post:
>> > > > > > >> http://www.open-mpi.org/community/lists/users/2014/08/25016.php
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > ---------------------------------
>> > > > > > Maxime Boissonneault
>> > > > > > Analyste de calcul - Calcul Qu饕ec, Universit・Laval
>> > > > > > Ph. D. en physique
>> > > > > >
>> > > > > > _______________________________________________
>> > > > > > users mailing list
>> > > > > > us...@open-mpi.org
>> > > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > > > > > Link to this post:
>> > > > > > http://www.open-mpi.org/community/lists/users/2014/08/25020.php
>> > > > > _______________________________________________
>> > > > > users mailing list
>> > > > > us...@open-mpi.org
>> > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > > > > Link to this post:
>> > > > > http://www.open-mpi.org/community/lists/users/2014/08/25032.php
>> > > >
>> > > > _______________________________________________
>> > > > users mailing list
>> > > > us...@open-mpi.org
>> > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > > > Link to this post:
>> > > > http://www.open-mpi.org/community/lists/users/2014/08/25034.php
>> > > _______________________________________________
>> > > users mailing list
>> > > us...@open-mpi.org
>> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > > Link to this post:
>> > > http://www.open-mpi.org/community/lists/users/2014/08/25037.php
>> >
>> > _______________________________________________
>> > users mailing list
>> > us...@open-mpi.org
>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> > Link to this post:
>> > http://www.open-mpi.org/community/lists/users/2014/08/25038.php
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/08/25079.php
>
>_______________________________________________
>users mailing list
>us...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post:
>http://www.open-mpi.org/community/lists/users/2014/08/25080.php
----
Tetsuya Mishima tmish...@jcity.maeda.co.jp
