Here's a better patch. Still only compile tested :)
Brice
Le 11/04/2012 10:36, Brice Goglin a écrit :
> A quick look at the code seems to confirm my feeling. get/set_module()
> callbacks manipulate arrays of logical indexes, and they do not convert
> them back to physical indexes before binding.
>
> Here's a quick patch that may help. Only compile tested...
>
> Brice
>
>
>
> Le 11/04/2012 09:49, Brice Goglin a écrit :
>> Le 11/04/2012 09:06, [email protected] a écrit :
>>> Hi, Brice.
>>>
>>> I installed the latest hwloc-1.4.1.
>>> Here is the output of lstopo -p.
>>>
>>> [root@node03 bin]# ./lstopo -p
>>> Machine (126GB)
>>> Socket P#0 (32GB)
>>> NUMANode P#0 (16GB) + L3 (5118KB)
>>> L2 (512KB) + L1 (64KB) + Core P#0 + PU P#0
>>> L2 (512KB) + L1 (64KB) + Core P#1 + PU P#4
>>> L2 (512KB) + L1 (64KB) + Core P#2 + PU P#8
>>> L2 (512KB) + L1 (64KB) + Core P#3 + PU P#12
>> Ok then the cpuset of this numanode is 1111.
>>
>>>> [node03.cluster:21706] [[55518,0],0] odls:default:fork binding child
>>>> [[55518,1],0] to cpus 1111
>> So openmpi 1.5.4 is correct.
>>
>>>> [node03.cluster:04706] [[40566,0],0] odls:default:fork binding child
>>>> [[40566,1],0] to cpus 000f
>> And openmpi 1.5.5 is indeed wrong.
>>
>> Random guess: 000f is the bitmask made of hwloc *logical* indexes. hwloc
>> cpusets (used for binding) are internally made of hwloc *physical*
>> indexes (1111 here).
>>
>> Jeff, Ralph:
>> How does OMPI 1.5.5 build hwloc cpusets for binding? Are you doing
>> bitmap operations on hwloc object cpusets?
>> If yes, I don't know what's going wrong here.
>> If no, are you building hwloc cpusets manually by setting individual
>> bits from object indexes? If yes, you must use *physical* indexes to do so.
>>
>> Brice
>>
>> _______________________________________________
>> users mailing list
>> [email protected]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> [email protected]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--- opal/mca/paffinity/hwloc/paffinity_hwloc_module.c.old 2012-04-11 10:19:36.766710073 +0200
+++ opal/mca/paffinity/hwloc/paffinity_hwloc_module.c 2012-04-11 11:13:52.930438083 +0200
@@ -164,9 +164,10 @@
static int module_set(opal_paffinity_base_cpu_set_t mask)
{
- int i, ret = OPAL_SUCCESS;
+ int ret = OPAL_SUCCESS;
hwloc_bitmap_t set;
hwloc_topology_t *t;
+ hwloc_obj_t pu;
/* bozo check */
if (NULL == opal_hwloc_topology) {
@@ -178,10 +179,11 @@
if (NULL == set) {
return OPAL_ERR_OUT_OF_RESOURCE;
}
- hwloc_bitmap_zero(set);
- for (i = 0; ((unsigned int) i) < OPAL_PAFFINITY_BITMASK_CPU_MAX; ++i) {
- if (OPAL_PAFFINITY_CPU_ISSET(i, mask)) {
- hwloc_bitmap_set(set, i);
+ for (pu = hwloc_get_obj_by_type(*t, HWLOC_OBJ_PU, 0);
+ pu && pu->logical_index < OPAL_PAFFINITY_BITMASK_CPU_MAX;
+ pu = pu->next_cousin) {
+ if (OPAL_PAFFINITY_CPU_ISSET(pu->logical_index, mask)) {
+ hwloc_bitmap_set(set, pu->os_index);
}
}
@@ -196,9 +198,10 @@
static int module_get(opal_paffinity_base_cpu_set_t *mask)
{
- int i, ret = OPAL_SUCCESS;
+ int ret = OPAL_SUCCESS;
hwloc_bitmap_t set;
hwloc_topology_t *t;
+ hwloc_obj_t pu;
/* bozo check */
if (NULL == opal_hwloc_topology) {
@@ -218,9 +221,11 @@
ret = OPAL_ERR_IN_ERRNO;
} else {
OPAL_PAFFINITY_CPU_ZERO(*mask);
- for (i = 0; ((unsigned int) i) < 8 * sizeof(*mask); i++) {
- if (hwloc_bitmap_isset(set, i)) {
- OPAL_PAFFINITY_CPU_SET(i, *mask);
+ for (pu = hwloc_get_obj_by_type(*t, HWLOC_OBJ_PU, 0);
+ pu && pu->logical_index < 8 * sizeof(*mask);
+ pu = pu->next_cousin) {
+ if (hwloc_bitmap_isset(set, pu->os_index)) {
+ OPAL_PAFFINITY_CPU_SET(pu->logical_index, *mask);
}
}
}
