Hi Mike,
I have similar configurations except a cluster of AMD-based linux
platforms with 2 GPU cards.
Your suggestion works. However, the performance of 2 GPU discourages
me because , for example, with 1 GPU, our computer node can easily
obtain a simulation of 31ns/day for a protein of 300 amino acids but
with 2 GPUs, it goes as far as 38 ns/day. I am very curious as to why
the performance of 2 GPUs is under expectation. Is there any overhead
that we should pay attention to ? Note that these 2GPU cards are
linked by a SLI bridge within the same node.
Since the computer nodes of our cluster have at least one GPU but
they are connected by slow network cards ( 1GB/sec), unfortunately, I
reasonably doubt that the performance will not be proportional to the
total number of GPU cards. I am wondering if you have any suggestion
about a cluster of GPU nodes. For example, will a infiniband
networking help increase a final performance when we execute a mpi
task ? or what else ? or forget about mpi and use single GPU instead.
Any suggestion is highly appreciated.
Thanks.
Dwey
> Date: Tue, 5 Nov 2013 16:20:39 +0100
> From: Mark Abraham <mark.j.abra...@gmail.com>
> Subject: Re: [gmx-users] Gromacs-4.6 on two Titans GPUs
> To: Discussion list for GROMACS users <gmx-users@gromacs.org>
> Message-ID:
> <camnumasm5ht40ub+unppv7gmhqzxsb6psewma+hblv+gnb2...@mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Tue, Nov 5, 2013 at 12:55 PM, James Starlight
> <jmsstarli...@gmail.com>wrote:
>
>> Dear Richard,
>>
>>
>> 1) mdrun -ntmpi 1 -ntomp 12 -gpu_id 0 -v -deffnm md_CaM_test
>> gave me performance about 25ns/day for the explicit solved system consisted
>> of 68k atoms (charmm ff. 1.0 cutoofs)
>>
>> gaves slightly worse performation in comparison to the 1)
>>
>>
> Richard suggested
>
> mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test,
>
> which looks correct to me. -ntomp 6 is probably superfluous
>
> Mark
>
>
>> finally
>>
>> 3) mdrun -deffnm md_CaM_test
>> running in the same regime as in the 2) so its also gave me 22ns/day for
>> the same system.
>>
>> How the efficacy of using of dual-GPUs could be increased?
>>
>> James
>>
>>
>> 2013/11/5 Richard Broadbent <richard.broadben...@imperial.ac.uk>
>>
>> > Dear James,
>> >
>> >
>> > On 05/11/13 11:16, James Starlight wrote:
>> >
>> >> My suggestions:
>> >>
>> >> 1) During compilstion using -march=corei7-avx-i I have obtained error
>> that
>> >> somethng now found ( sorry I didnt save log) so I compile gromacs
>> without
>> >> this flag
>> >>
>> >> 2) I have twice as better performance using just 1 gpu by means of
>> >>
>> >> mdrun -ntmpi 1 -ntomp 12 -gpu_id 0 -v -deffnm md_CaM_test
>> >>
>> >> than using of both gpus
>> >>
>> >> mdrun -ntmpi 2 -ntomp 12 -gpu_id 01 -v -deffnm md_CaM_test
>> >>
>> >> in the last case I have obtained warning
>> >>
>> >> WARNING: Oversubscribing the available 12 logical CPU cores with 24
>> >> threads.
>> >> This will cause considerable performance loss!
>> >>
>> >> here you are requesting 2 thread mpi processes each with 12 openmp
>> > threads, hence a total of 24 threads however even with hyper threading
>> > enabled there are only 12 threads on your machine. Therefore, only
>> allocate
>> > 12. Try
>> >
>> > mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v -deffnm md_CaM_test
>> >
>> > or even
>> >
>> > mdrun -v -deffnm md_CaM_test
>> >
>> > I believe it should autodetect the GPUs and run accordingly for details
>> of
>> > how to use gromacs with mpi/thread mpi openmp and GPUs see
>> >
>> > http://www.gromacs.org/Documentation/Acceleration_and_parallelization
>> >
>> > Which describes how to use these systems
>> >
>> > Richard
>> >
>> >
>> > How it could be fixed?
>> >> All gpu are recognized correctly
>> >>
>> >>
>> >> 2 GPUs detected:
>> >> #0: NVIDIA GeForce GTX TITAN, compute cap.: 3.5, ECC: no, stat:
>> >> compatible
>> >> #1: NVIDIA GeForce GTX TITAN, compute cap.: 3.5, ECC: no, stat:
>> >> compatible
>> >>
>> >>
>> >> James
>> >>
>> >>
>> >> 2013/11/4 Szilárd Páll <pall.szil...@gmail.com>
>> >>
>> >> You can use the "-march=native" flag with gcc to optimize for the CPU
>> >>> your are building on or e.g. -march=corei7-avx-i for Intel Ivy Bridge
>> >>> CPUs.
>> >>> --
>> >>> Szilárd Páll
>> >>>
>> >>>
>> >>> On Mon, Nov 4, 2013 at 12:37 PM, James Starlight <
>> jmsstarli...@gmail.com
>> >>> >
>> >>> wrote:
>> >>>
>> >>>> Szilárd, thanks for suggestion!
>> >>>>
>> >>>> What kind of CPU optimisation should I take into account assumint that
>> >>>>
>> >>> I'm
>> >>>
>> >>>> using dual-GPU Nvidia TITAN workstation with 6 cores i7 (recognized as
>> >>>> 12
>> >>>> nodes in Debian).
>> >>>>
>> >>>> James
>> >>>>
>> >>>>
>> >>>> 2013/11/4 Szilárd Páll <pall.szil...@gmail.com>
>> >>>>
>> >>>> That should be enough. You may want to use the -march (or equivalent)
>> >>>>> compiler flag for CPU optimization.
>> >>>>>
>> >>>>> Cheers,
>> >>>>> --
>> >>>>> Szilárd Páll
>> >>>>>
>> >>>>>
>> >>>>> On Sun, Nov 3, 2013 at 10:01 AM, James Starlight <
>> >>>>>
>> >>>> jmsstarli...@gmail.com>
>> >>>
>> >>>> wrote:
>> >>>>>
>> >>>>>> Dear Gromacs Users!
>> >>>>>>
>> >>>>>> I'd like to compile lattest 4.6 Gromacs with native GPU supporting
>> on
>> >>>>>>
>> >>>>> my
>> >>>
>> >>>> i7
>> >>>>>
>> >>>>>> cpu with dual GeForces Titans gpu mounted. With this config I'd like
>> >>>>>>
>> >>>>> to
>> >>>
>> >>>> perform simulations using cpu as well as both gpus simultaneously.
>> >>>>>>
>> >>>>>> What flags besides
>> >>>>>>
>> >>>>>> cmake .. -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-5.5
>> >>>>>>
>> >>>>>>
>> >>>>>> should I define to CMAKE for compiling optimized gromacs on such
>> >>>>>>
>> >>>>> workstation?
>> >>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Thanks for help
>> >>>>>>
>> >>>>>> James
>> >>>>>> --
>> >>>>>> gmx-users mailing list gmx-users@gromacs.org
>> >>>>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> >>>>>> * Please search the archive at
>> >>>>>>
>> >>>>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> >>>>>
>> >>>>>> * Please don't post (un)subscribe requests to the list. Use the
>> >>>>>> www interface or send it to gmx-users-requ...@gromacs.org.
>> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>>>>
>> >>>>> --
>> >>>>> gmx-users mailing list gmx-users@gromacs.org
>> >>>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> >>>>> * Please search the archive at
>> >>>>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> >>>>> * Please don't post (un)subscribe requests to the list. Use the
>> >>>>> www interface or send it to gmx-users-requ...@gromacs.org.
>> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>>>
>> >>>>> --
>> >>>> gmx-users mailing list gmx-users@gromacs.org
>> >>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> >>>> * Please search the archive at
>> >>>>
>> >>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> >>>
>> >>>> * Please don't post (un)subscribe requests to the list. Use the
>> >>>> www interface or send it to gmx-users-requ...@gromacs.org.
>> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>>
>> >>> --
>> >>> gmx-users mailing list gmx-users@gromacs.org
>> >>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> >>> * Please search the archive at
>> >>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> >>> * Please don't post (un)subscribe requests to the list. Use the
>> >>> www interface or send it to gmx-users-requ...@gromacs.org.
>> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>
>> >>> --
>> > gmx-users mailing list gmx-users@gromacs.org
>> > http://lists.gromacs.org/mailman/listinfo/gmx-users
>> > * Please search the archive at http://www.gromacs.org/
>> > Support/Mailing_Lists/Search before posting!
>> > * Please don't post (un)subscribe requests to the list. Use the www
>> > interface or send it to gmx-users-requ...@gromacs.org.
>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >
>> --
>> gmx-users mailing list gmx-users@gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> * Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-users-requ...@gromacs.org.
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>
>
> ------------------------------
>
> --
> gmx-users mailing list
> gmx-users@gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-users
> Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>
> End of gmx-users Digest, Vol 115, Issue 22
> ******************************************
--
gmx-users mailing list gmx-users@gromacs.org
http://lists.gromacs.org/mailman/listinfo/gmx-users
* Please search the archive at
http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
* Please don't post (un)subscribe requests to the list. Use the
www interface or send it to gmx-users-requ...@gromacs.org.
* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
