Hello Tetsuya, I think it's expected that the displayed cpusets are different. I only converted the code that applies/retrieves the binding, I did not touch the code that prints it. Good to know it works. Brice
Le 12/04/2012 05:24, tmish...@jcity.maeda.co.jp a écrit : > Hi, Brice. > > Thank you for sending me a patch. Now, I quickly tested your try2.patch. > > Then, regarding execution speed it works well. > But, in terms of core binding reports, it's still different from > openmpi-1.5.4. > I'm not sure which is better for a standard user like me, reporting logical > indexes or physical ones. > > patched openmpi-1.5.5 Reports: > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],1] to cpus 00f0 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],2] to cpus 0f00 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],3] to cpus f000 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],4] to cpus f0000 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],5] to cpus f00000 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],6] to cpus f000000 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],7] to cpus f0000000 > [node03.cluster:09780] [[43552,0],0] odls:default:fork binding child > [[43552,1],0] to cpus 000f > > Regards, > Tetsuya Mishima > >> Here's a better patch. Still only compile tested :) >> Brice >> >> >> Le 11/04/2012 10:36, Brice Goglin a écrit : >> >> A quick look at the code seems to confirm my feeling. get/set_module() >> callbacks manipulate arrays of logical indexes, and they do not convert >> them back to physical indexes before binding. >> >> Here's a quick patch that may help. Only compile tested... >> >> Brice >> >> >> >> Le 11/04/2012 09:49, Brice Goglin a écrit : >> >> Le 11/04/2012 09:06, tmish...@jcity.maeda.co.jp a écrit : >> >> Hi, Brice. >> >> I installed the latest hwloc-1.4.1. >> Here is the output of lstopo -p. >> >> [root@node03 bin]# ./lstopo -p >> Machine (126GB) >> Socket P#0 (32GB) >> NUMANode P#0 (16GB) + L3 (5118KB) >> L2 (512KB) + L1 (64KB) + Core P#0 + PU P#0 >> L2 (512KB) + L1 (64KB) + Core P#1 + PU P#4 >> L2 (512KB) + L1 (64KB) + Core P#2 + PU P#8 >> L2 (512KB) + L1 (64KB) + Core P#3 + PU P#12 >> >> Ok then the cpuset of this numanode is 1111. >> >> [node03.cluster:21706] [[55518,0],0] odls:default:fork binding child >> [[55518,1],0] to cpus 1111 >> >> So openmpi 1.5.4 is correct. >> >> [node03.cluster:04706] [[40566,0],0] odls:default:fork binding child >> [[40566,1],0] to cpus 000f >> >> And openmpi 1.5.5 is indeed wrong. >> >> Random guess: 000f is the bitmask made of hwloc *logical* indexes. hwloc >> cpusets (used for binding) are internally made of hwloc *physical* >> indexes (1111 here). >> >> Jeff, Ralph: >> How does OMPI 1.5.5 build hwloc cpusets for binding? Are you doing >> bitmap operations on hwloc object cpusets? >> If yes, I don't know what's going wrong here. >> If no, are you building hwloc cpusets manually by setting individual >> bits from object indexes? If yes, you must use *physical* indexes to do > so. >> Brice >> >> _______________________________________________ >> users mailing > listusers@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing > listusers@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> --- opal/mca/paffinity/hwloc/paffinity_hwloc_module.c.old 2012-04-11 > 10:19:36.766710073 +0200 >> +++ opal/mca/paffinity/hwloc/paffinity_hwloc_module.c 2012-04-11 > 11:13:52.930438083 +0200 >> @@ -164,9 +164,10 @@ >> >> static int module_set(opal_paffinity_base_cpu_set_t mask) >> { >> - int i, ret = OPAL_SUCCESS; >> + int ret = OPAL_SUCCESS; >> hwloc_bitmap_t set; >> hwloc_topology_t *t; >> + hwloc_obj_t pu; >> >> /* bozo check */ >> if (NULL == opal_hwloc_topology) { >> @@ -178,10 +179,11 @@ >> if (NULL == set) { >> return OPAL_ERR_OUT_OF_RESOURCE; >> } >> - hwloc_bitmap_zero(set); >> - for (i = 0; ((unsigned int) i) < OPAL_PAFFINITY_BITMASK_CPU_MAX; + > +i) { >> - if (OPAL_PAFFINITY_CPU_ISSET(i, mask)) { >> - hwloc_bitmap_set(set, i); >> + for (pu = hwloc_get_obj_by_type(*t, HWLOC_OBJ_PU, 0); >> + pu && pu->logical_index < OPAL_PAFFINITY_BITMASK_CPU_MAX; >> + pu = pu->next_cousin) { >> + if (OPAL_PAFFINITY_CPU_ISSET(pu->logical_index, mask)) { >> + hwloc_bitmap_set(set, pu->os_index); >> } >> } >> >> @@ -196,9 +198,10 @@ >> >> static int module_get(opal_paffinity_base_cpu_set_t *mask) >> { >> - int i, ret = OPAL_SUCCESS; >> + int ret = OPAL_SUCCESS; >> hwloc_bitmap_t set; >> hwloc_topology_t *t; >> + hwloc_obj_t pu; >> >> /* bozo check */ >> if (NULL == opal_hwloc_topology) { >> @@ -218,9 +221,11 @@ >> ret = OPAL_ERR_IN_ERRNO; >> } else { >> OPAL_PAFFINITY_CPU_ZERO(*mask); >> - for (i = 0; ((unsigned int) i) < 8 * sizeof(*mask); i++) { >> - if (hwloc_bitmap_isset(set, i)) { >> - OPAL_PAFFINITY_CPU_SET(i, *mask); >> + for (pu = hwloc_get_obj_by_type(*t, HWLOC_OBJ_PU, 0); >> + pu && pu->logical_index < 8 * sizeof(*mask); >> + pu = pu->next_cousin) { >> + if (hwloc_bitmap_isset(set, pu->os_index)) { >> + OPAL_PAFFINITY_CPU_SET(pu->logical_index, *mask); >> } >> } >> } >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
