On Mon, Jun 16, 2008 at 9:46 PM, Brock Palen <bro...@umich.edu> wrote:
> Brad just curious. > Did you tweak any other values for starting and running a job on such a > large system? You say unmodified, but OpenMPI lets you tweak many values > at runtime. > Ahh...sorry for the confusion. By "unmodified", I meant at the source code level. The version of Open MPI used was from a straight checkout of the trunk and did not have any special code modifications for Roadrunner. We certainly did change some of the default run-time settings. > I would be curious to expand what I know from what you discovered. > The main run-time parameters used are as follows: - For scalable process launch routed = binomial - To reduce the overhead for IB connections: btl_openib_receive_queues = P,128,64,32,32,32:S,2048,1024,128,32:S,12288,1024,128,32:S,65536,1024,128,32 - Bind processes to computational cores: opal_paffinity_alone = 1 - Leave buffers pinned and registered mpi_leave_pinned = 1 --brad > > Brock Palen > www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp> > Center for Advanced Computing > bro...@umich.edu > (734)936-1985 > > > > On Jun 16, 2008, at 10:12 PM, Brad Benton wrote: > > Greetings Open MPI users; we thought you'd be interested in the > following announcement... > > A new supercomputer, powered by Open MPI, has broken the petaflop > barrier to become the world's fastest supercomputer. The > "Roadrunner" system was jointly developed by Los Alamos National > Laboratories and IBM. Roadrunner's design uses a cluster of AMD > dual-core processors coupled with computational accelerators based > on the IBM Cell Broadband Engine. The cluster consists of 3,060 > nodes, each of which has 2 dual-core AMD processors and associated > Cell accelerators. The AMD nodes are connected with 4x DDR > InfiniBand links. > > Open MPI was used as the communications library for the 12,240 > processes comprising the Linpack run which broke the Petaflop > barrier at 1.026 Petaflop/s. The version of Open MPI used in the > run-for-record was a pre-release version of the upcoming 1.3 > release. Enhancements in this release include modifications for > efficient, scalable process launch. As such, Open MPI was run > unmodified from a snapshot of the pre-1.3 source base (meaning: > there are no Roadrunner-specific enhancements that are unportable to > other environments -- all Open MPI users benefit from the > scalability and performance improvements contributed by the > Roadrunner project). > > --Brad Benton > Open MPI/Roadrunner Team > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
