Thank you, Mark and Xavier.
The thing is that the cluster manager set the minimum number of cores of each jobs in Bluegene/Q is 128, so I can not use 64 cores. But according to the performance, 512 cores in Bluegene roughly equivalent to 64 cores in another cluster. Since there are 16 cores in each computational cards, the total number of cores I used in Bluegene//Q is num_cards times 16. So in my test, I acutally run simulations using different number of cards, from 8 to 256. The following is the script I submitted to bluegene using 128 computational cards: #!/bin/sh #SBATCH --nodes=128 # set Use 128 Compute Cards ( 1x Compute Card = 16 cores, 128x16 = 2048 cores ) #SBATCH --job-name="128x16x2" # set Job name #SBATCH -output="first-job-sample" # set Output file #SBATCH --partition="training" srun --ntasks-per-node=32 --overcommit /scratch/home/biilijg/package/gromacs-461/bin/mdrun -s box_md1.tpr -c box_md1.gro -x box_md1.xtc -g md1.log >& job_md1 Since bluegene/q accepts up to 4 tasks each core, I used 32 mpi tasks for each card (2 task per core). I tried --ntasks-per-node=64, but the simulations get much slower. Is there a optimized number for --ntasks-per-node? Cheers Jianguo ________________________________ From: Mark Abraham <mark.j.abra...@gmail.com> To: Discussion list for GROMACS users <gmx-users@gromacs.org> Cc: Jianguo Li <ljg...@yahoo.com.sg> Sent: Tuesday, 4 June 2013, 22:32 Subject: Re: [gmx-users] Performance of Gromacs-4.6.1 on BlueGene/Q On Tue, Jun 4, 2013 at 4:20 PM, XAvier Periole <x.peri...@rug.nl> wrote: >BG CPUs are generally much slower (clock whose) but scale better. > >You should try to run on 64 CPUs on the Blue gene too for faire comparison. >The number of CPUs per nodes is also an important factor: the more CPUs per >nodes the more communications needs to be done. I observed a significant slow >down while going from 16 to 32 CPUs nodes (recent intel) but using the same >number of CPUs. > Indeed. Moreover, there is not (yet) any instruction-level parallelism in the GROMACS kernels used on BG/Q, unlike for the x86 family. So there is a theoretical factor of four that is simply not being exploited. (And no, the compiler is not good enough to do it automatically ;-)) Mark >On Jun 4, 2013, at 4:02 PM, Jianguo Li <ljg...@yahoo.com.sg> wrote: > >> Dear All, >> >> >> Has anyone has Gromacs benchmark on Bluegene/Q? >> I recently installed gromacs-461 on BG/Q using the following command: >> cmake .. -DCMAKE_TOOLCHAIN_FILE=BlueGeneQ-static-XL-C \ >> -DGMX_BUILD_OWN_FFTW=ON \ >> -DBUILD_SHARED_LIBS=OFF \ >> -DGMX_XML=OFF \ >> -DCMAKE_INSTALL_PREFIX=/scratch/home/biilijg/package/gromacs-461 >> make >> make install >> >> After that, I did a benchmark simulation using a box of pure water >> containing 140k atoms. >> The command I used for the above test is: >> srun --ntasks-per-node=32 --overcommit >> /scratch/home/biilijg/package/gromacs-461/bin/mdrun -s box_md1.tpr -c >> box_md1.gro -x box_md1.xtc -g md1.log >& job_md1 >> >> And I got the following performance: >> Num. cores hour/ns >> 128 9.860 >> 256 4.984 >> 512 2.706 >> 1024 1.544 >> 2048 0.978 >> 4092 0.677 >> >> The scaling seems ok, but the performance is far from what I expected. In >> terms CPU-to-CPU performance, the Bluegene is 8 times slower than other >> clusters. For comparison, I also did the same simulation using 64 processors >> in a SGI cluster, and I got 2.8 hour/ns, which is roughly equivalent to >> using 512 cores in BlueGene/Q. >> >> I am wondering if the above benchmark results are reasonable or not? Or Am I >> doing something wrong in compiling? >> Any comments/suggestions are appreciated, thank you very much! >> >> Have a nice day! >> Jianguo >> >> -- >> gmx-users mailing list gmx-users@gromacs.org >> http://lists.gromacs.org/mailman/listinfo/gmx-users >> * Please search the archive at >> http://www.gromacs.org/Support/Mailing_Lists/Search before posting! >> * Please don't post (un)subscribe requests to the list. Use the >> www interface or send it to gmx-users-requ...@gromacs.org. >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >-- >gmx-users mailing list gmx-users@gromacs.org >http://lists.gromacs.org/mailman/listinfo/gmx-users >* Please search the archive at >http://www.gromacs.org/Support/Mailing_Lists/Search before posting! >* Please don't post (un)subscribe requests to the list. Use the >www interface or send it to gmx-users-requ...@gromacs.org. >* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > -- gmx-users mailing list gmx-users@gromacs.org http://lists.gromacs.org/mailman/listinfo/gmx-users * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! * Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-users-requ...@gromacs.org. * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
