Ralph filed https://svn.open-mpi.org/trac/ompi/ticket/3075 about this.
I just reviewed Brice's patch and it looks good to me (it looks like I forgot to update the logical/physical conversion in the module_set/get methods). Thanks! On Apr 12, 2012, at 1:46 AM, Brice Goglin wrote: > Hello Tetsuya, > I think it's expected that the displayed cpusets are different. > I only converted the code that applies/retrieves the binding, I did not > touch the code that prints it. > Good to know it works. > Brice > > > > Le 12/04/2012 05:24, tmish...@jcity.maeda.co.jp a écrit : >> Hi, Brice. >> >> Thank you for sending me a patch. Now, I quickly tested your try2.patch. >> >> Then, regarding execution speed it works well. >> But, in terms of core binding reports, it's still different from >> openmpi-1.5.4. >> I'm not sure which is better for a standard user like me, reporting logical >> indexes or physical ones. >> >> patched openmpi-1.5.5 Reports: >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],1] to cpus 00f0 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],2] to cpus 0f00 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],3] to cpus f000 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],4] to cpus f0000 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],5] to cpus f00000 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],6] to cpus f000000 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],7] to cpus f0000000 >> [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child >> [[43552,1],0] to cpus 000f >> >> Regards, >> Tetsuya Mishima >> >>> Here's a better patch. Still only compile tested :) >>> Brice >>> >>> >>> Le 11/04/2012 10:36, Brice Goglin a écrit : >>> >>> A quick look at the code seems to confirm my feeling. get/set_module() >>> callbacks manipulate arrays of logical indexes, and they do not convert >>> them back to physical indexes before binding. >>> >>> Here's a quick patch that may help. Only compile tested... >>> >>> Brice >>> >>> >>> >>> Le 11/04/2012 09:49, Brice Goglin a écrit : >>> >>> Le 11/04/2012 09:06, tmish...@jcity.maeda.co.jp a écrit : >>> >>> Hi, Brice. >>> >>> I installed the latest hwloc-1.4.1. >>> Here is the output of lstopo -p. >>> >>> [root@node03 bin]# ./lstopo -p >>> Machine (126GB) >>> Socket P#0 (32GB) >>> NUMANode P#0 (16GB) + L3 (5118KB) >>> L2 (512KB) + L1 (64KB) + Core P#0 + PU P#0 >>> L2 (512KB) + L1 (64KB) + Core P#1 + PU P#4 >>> L2 (512KB) + L1 (64KB) + Core P#2 + PU P#8 >>> L2 (512KB) + L1 (64KB) + Core P#3 + PU P#12 >>> >>> Ok then the cpuset of this numanode is 1111. >>> >>> [node03.cluster:21706] [[55518,0],0] odls:default:fork binding child >>> [[55518,1],0] to cpus 1111 >>> >>> So openmpi 1.5.4 is correct. >>> >>> [node03.cluster:04706] [[40566,0],0] odls:default:fork binding child >>> [[40566,1],0] to cpus 000f >>> >>> And openmpi 1.5.5 is indeed wrong. >>> >>> Random guess: 000f is the bitmask made of hwloc *logical* indexes. hwloc >>> cpusets (used for binding) are internally made of hwloc *physical* >>> indexes (1111 here). >>> >>> Jeff, Ralph: >>> How does OMPI 1.5.5 build hwloc cpusets for binding? Are you doing >>> bitmap operations on hwloc object cpusets? >>> If yes, I don't know what's going wrong here. >>> If no, are you building hwloc cpusets manually by setting individual >>> bits from object indexes? If yes, you must use *physical* indexes to do >> so. >>> Brice >>> >>> _______________________________________________ >>> users mailing >> listusers@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> _______________________________________________ >>> users mailing >> listusers@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> --- opal/mca/paffinity/hwloc/paffinity_hwloc_module.c.old 2012-04-11 >> 10:19:36.766710073 +0200 >>> +++ opal/mca/paffinity/hwloc/paffinity_hwloc_module.c 2012-04-11 >> 11:13:52.930438083 +0200 >>> @@ -164,9 +164,10 @@ >>> >>> static int module_set(opal_paffinity_base_cpu_set_t mask) >>> { >>> - int i, ret = OPAL_SUCCESS; >>> + int ret = OPAL_SUCCESS; >>> hwloc_bitmap_t set; >>> hwloc_topology_t *t; >>> + hwloc_obj_t pu; >>> >>> /* bozo check */ >>> if (NULL == opal_hwloc_topology) { >>> @@ -178,10 +179,11 @@ >>> if (NULL == set) { >>> return OPAL_ERR_OUT_OF_RESOURCE; >>> } >>> - hwloc_bitmap_zero(set); >>> - for (i = 0; ((unsigned int) i) < OPAL_PAFFINITY_BITMASK_CPU_MAX; + >> +i) { >>> - if (OPAL_PAFFINITY_CPU_ISSET(i, mask)) { >>> - hwloc_bitmap_set(set, i); >>> + for (pu = hwloc_get_obj_by_type(*t, HWLOC_OBJ_PU, 0); >>> + pu && pu->logical_index < OPAL_PAFFINITY_BITMASK_CPU_MAX; >>> + pu = pu->next_cousin) { >>> + if (OPAL_PAFFINITY_CPU_ISSET(pu->logical_index, mask)) { >>> + hwloc_bitmap_set(set, pu->os_index); >>> } >>> } >>> >>> @@ -196,9 +198,10 @@ >>> >>> static int module_get(opal_paffinity_base_cpu_set_t *mask) >>> { >>> - int i, ret = OPAL_SUCCESS; >>> + int ret = OPAL_SUCCESS; >>> hwloc_bitmap_t set; >>> hwloc_topology_t *t; >>> + hwloc_obj_t pu; >>> >>> /* bozo check */ >>> if (NULL == opal_hwloc_topology) { >>> @@ -218,9 +221,11 @@ >>> ret = OPAL_ERR_IN_ERRNO; >>> } else { >>> OPAL_PAFFINITY_CPU_ZERO(*mask); >>> - for (i = 0; ((unsigned int) i) < 8 * sizeof(*mask); i++) { >>> - if (hwloc_bitmap_isset(set, i)) { >>> - OPAL_PAFFINITY_CPU_SET(i, *mask); >>> + for (pu = hwloc_get_obj_by_type(*t, HWLOC_OBJ_PU, 0); >>> + pu && pu->logical_index < 8 * sizeof(*mask); >>> + pu = pu->next_cousin) { >>> + if (hwloc_bitmap_isset(set, pu->os_index)) { >>> + OPAL_PAFFINITY_CPU_SET(pu->logical_index, *mask); >>> } >>> } >>> } >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
