Okay, we've dug a bit and talked a lot :-) If you want to do this with the default mapping system, just add "--use-hwthreads-cpus" so we use the hwthreads as independent processors. Otherwise, you can only bind down to the core level since we aren't treating the HTs inside each core as separate entities.
On Sep 12, 2012, at 6:50 AM, Jeff Squyres <jsquy...@cisco.com> wrote: > A little more on this (since affinity is one of my favorite topics of late > :-) ). See my blog entries about what we just did in the 1.7 branch (and SVN > trunk): > > http://blogs.cisco.com/performance/taking-mpi-process-affinity-to-the-next-level/ > http://blogs.cisco.com/performance/process-affinity-in-ompi-v1-7-part-1/ > http://blogs.cisco.com/performance/process-affinity-in-ompi-v1-7-part-2/ > > As Ralph said, the v1.6 series will allow you to bind processes to the entire > core (i.e., all hyperthreads in a core), or to an entire socket (i.e., all > hyperthreads in a socket). > > The v1.7 series will be quite a bit more flexible in its affinity options > (note that the "Expert" mode described in my blog posting will be coming in > v1.7.1 -- if you want to try that now, you'll need to use the SVN trunk). > > For example: > > mpirun --report-bindings --map-by core --bind-to hwthread ... > > Should give you the pattern you want. Note that it looks like we have a bug > in this pattern at the moment, however -- you'll need to use the SVN trunk > and the "lama" mapper to get the patterns you want. The following example is > running on a sandy bridge server with 2 sockets, each with 8 cores, each with > 2 hyperthreads: > > One hyperthread per core: > > ----- > % mpirun --mca rmaps lama -np 4 --host svbu-mpi058 --report-bindings --map-by > core --bind-to hwthread uptime > [svbu-mpi058:23916] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > [B./../../../../../../..][../../../../../../../..] > [svbu-mpi058:23916] MCW rank 1 bound to socket 0[core 1[hwt 0]]: > [../B./../../../../../..][../../../../../../../..] > [svbu-mpi058:23916] MCW rank 2 bound to socket 0[core 2[hwt 0]]: > [../../B./../../../../..][../../../../../../../..] > [svbu-mpi058:23916] MCW rank 3 bound to socket 0[core 3[hwt 0]]: > [../../../B./../../../..][../../../../../../../..] > 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05 > 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05 > 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05 > 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05 > ----- > > Both hyperthreads per core: > > ----- > % mpirun --mca rmaps lama -np 4 --host svbu-mpi058 --report-bindings --map-by > core --bind-to core uptime > [svbu-mpi058:23951] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: > [BB/../../../../../../..][../../../../../../../..] > [svbu-mpi058:23951] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: > [../BB/../../../../../..][../../../../../../../..] > [svbu-mpi058:23951] MCW rank 2 bound to socket 0[core 2[hwt 0-1]]: > [../../BB/../../../../..][../../../../../../../..] > [svbu-mpi058:23951] MCW rank 3 bound to socket 0[core 3[hwt 0-1]]: > [../../../BB/../../../..][../../../../../../../..] > 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05 > 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05 > 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05 > 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05 > ------ > > > > On Sep 12, 2012, at 8:10 AM, John R. Cary wrote: > >> Thanks! >> >> John >> >> On 9/12/12 8:05 AM, Ralph Castain wrote: >>> On Sep 12, 2012, at 4:57 AM, "John R. Cary" <c...@txcorp.com> wrote: >>> >>>> I do want in fact to bind first to one HT of each core >>>> before binding to two HTs of one core. So that will >>>> be possible in 1.7? >>> Yes - you can get a copy of the 1.7 nightly tarball and experiment with it >>> in advance, if you like. You'll want >>> >>> mpirun --map-by core --bind-to hwthread .... >>> >>> Add --report-bindings to see what happens, but I believe that will do what >>> you want. You'll map one process to each core, and bind it to only the >>> first hwthread on that core. >>> >>> Let me know either way - if it doesn't, we have time to adjust it. >>> >>>> Thx....John >>>> >>>> On 9/11/12 11:19 PM, Ralph Castain wrote: >>>>> Not entirely sure I know what you mean. If you are talking about running >>>>> without specifying binding, then it makes no difference - we'll run >>>>> wherever the OS puts us, so you would need to tell the OS not to use the >>>>> virtual cores (i.e., disable HT). >>>>> >>>>> If you are talking about binding, then pre-1.7 releases all bind to core >>>>> at the lowest level. On a hyperthread-enabled machine, that binds you to >>>>> both HT's of a core. Starting with the upcoming 1.7 release, you can bind >>>>> to the separate HTs, but that doesn't sound like something you want to do. >>>>> >>>>> HTH >>>>> Ralph >>>>> >>>>> >>>>> On Sep 11, 2012, at 6:34 PM, John R. Cary <c...@txcorp.com> wrote: >>>>> >>>>>> Our code gets little benefit from using virtual cores (hyperthreading), >>>>>> so when we run with mpiexec on an 8 real plus 8 virtual machine, we >>>>>> would like to be certain that it uses only the 8 real cores. >>>>>> >>>>>> Is there a way to do this with openmpi? >>>>>> >>>>>> Thx....John >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
