Hi Ralph, I'm glad to hear that, thanks.
By the way, yesterday I tried to check how lama in 1.7.4rc treat numa node. Then, even wiht this simple command line, it freezed without any massage: mpirun -np 2 -host node05 -mca rmaps lama myprog Could you check what happened? Is it better to open new thread or continue this thread? Regards, Tetsuya Mishima > I'll make it work so that NUMA can be either above or below socket > > On Dec 20, 2013, at 2:57 AM, tmish...@jcity.maeda.co.jp wrote: > > > > > > > Hi Brice, > > > > Thank you for your comment. I understand what you mean. > > > > My opinion was made just considering easy way to adjust the code for > > inversion of hierarchy in object tree. > > > > Tetsuya Mishima > > > > > >> I don't think there's any such difference. > >> Also, all these NUMA architectures are reported the same by hwloc, and > >> therefore used the same in Open MPI. > >> > >> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours (and > >> most recent AMD and Intel platforms). > >> > >> Brice > >> > >> > >> > >> Le 20/12/2013 11:33, tmish...@jcity.maeda.co.jp a écrit : > >>> > >>> Hi Ralph, > >>> > >>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache > >>> coherent)NUMA, > >>> which seems to be a little bit different from the traditional numa > > defined > >>> in openmpi. > >>> > >>> I notice that ccNUMA object is almost same as L3cache object. > >>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want to > > do. > >>> Therefore, "do not touch it" is one of the solution, I think ... > >>> > >>> Anyway, mixing up these two types of numa is the problem. > >>> > >>> Regards, > >>> Tetsuya Mishima > >>> > >>>> I can wait it'll be fixed in 1.7.5 or later, because putting "-bind-to > >>>> numa" > >>>> and "-map-by numa" at the same time works as a workaround. > >>>> > >>>> Thanks, > >>>> Tetsuya Mishima > >>>> > >>>>> Yeah, it will impact everything that uses hwloc topology maps, I > > fear. > >>>>> > >>>>> One side note: you'll need to add --hetero-nodes to your cmd line. If > >>> we > >>>> don't see that, we assume that all the node topologies are identical - > >>>> which clearly isn't true here. > >>>>> I'll try to resolve the hier inversion over the holiday - won't be > > for > >>>> 1.7.4, but hopefully for 1.7.5 > >>>>> Thanks > >>>>> Ralph > >>>>> > >>>>> On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>> > >>>>>> > >>>>>> I think it's normal for AMD opteron having 8/16 cores such as > >>>>>> magny cours or interlagos. Because it usually has 2 numa nodes > >>>>>> in a cpu(socket), numa-node can not include a socket. This type > >>>>>> of hierarchy would be natural. > >>>>>> > >>>>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess) > >>>>>> > >>>>>> By the way, I think this inversion should affect rmaps_lama mapping. > >>>>>> > >>>>>> Tetsuya Mishima > >>>>>> > >>>>>>> Ick - yeah, that would be a problem. I haven't seen that type of > >>>>>> hierarchical inversion before - is node03 a different type of chip? > >>>>>>> Might take awhile for me to adjust the code to handle hier > >>>>>> inversion... :-( > >>>>>>> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> Hi Ralph, > >>>>>>>> > >>>>>>>> I found the reason. I attached the main part of output with 32 > >>>>>>>> core node(node03) and 8 core node(node05) at the bottom. > >>>>>>>> > >>>>>>>> From this information, socket of node03 includes numa-node. > >>>>>>>> On the other hand, numa-node of node05 includes socket. > >>>>>>>> The direction of object tree is opposite. > >>>>>>>> > >>>>>>>> Since "-map-by socket" may be assumed as default, > >>>>>>>> for node05, "-bind-to numa and -map-by socket" means > >>>>>>>> upward search. For node03, this should be downward. > >>>>>>>> > >>>>>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node > >>>>>>>> includes socket. Is it right? Then, upward search is assumed > >>>>>>>> in orte_rmaps_base_compute_bindings even for node03 when I > >>>>>>>> put "-bind-to numa and -map-by socket" option. > >>>>>>>> > >>>>>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage > >>>>>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job > >>> [38286,1] > >>>>>> with > >>>>>>>> policy NUMA > >>>>>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job [38286,1] > >>> with > >>>>>>>> bindings NUMA > >>>>>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> Machine > >>>>>>>> > >>>>>>>> That's the reason of this trouble. Therefore, adding "-map-by > > core" > >>>>>> works. > >>>>>>>> (mapping pattern seems to be strange ...) > >>>>>>>> > >>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by core > >>>>>>>> -report-bindings myprog > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>> Cache > >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>> type > >>>>>>>> NUMANode > >>>>>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>> > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>> > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>> > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>> > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>> > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>> > >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>> > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt > > 0]], > >>>>>> socket > >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>> > >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>> Hello world from process 6 of 8 > >>>>>>>> Hello world from process 5 of 8 > >>>>>>>> Hello world from process 0 of 8 > >>>>>>>> Hello world from process 7 of 8 > >>>>>>>> Hello world from process 3 of 8 > >>>>>>>> Hello world from process 4 of 8 > >>>>>>>> Hello world from process 2 of 8 > >>>>>>>> Hello world from process 1 of 8 > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Tetsuya Mishima > >>>>>>>> > >>>>>>>> [node03.cluster:15508] Type: Machine Number of child objects: 4 > >>>>>>>> Name=NULL > >>>>>>>> total=132358820KB > >>>>>>>> Backend=Linux > >>>>>>>> OSName=Linux > >>>>>>>> OSRelease=2.6.18-308.16.1.el5 > >>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >>>>>>>> Architecture=x86_64 > >>>>>>>> Cpuset: 0xffffffff > >>>>>>>> Online: 0xffffffff > >>>>>>>> Allowed: 0xffffffff > >>>>>>>> Bind CPU proc: TRUE > >>>>>>>> Bind CPU thread: TRUE > >>>>>>>> Bind MEM proc: FALSE > >>>>>>>> Bind MEM thread: TRUE > >>>>>>>> Type: Socket Number of child objects: 2 > >>>>>>>> Name=NULL > >>>>>>>> total=33071780KB > >>>>>>>> CPUModel="AMD Opteron(tm) Processor 6136" > >>>>>>>> Cpuset: 0x000000ff > >>>>>>>> Online: 0x000000ff > >>>>>>>> Allowed: 0x000000ff > >>>>>>>> Type: NUMANode Number of child objects: 1 > >>>>>>>> > >>>>>>>> > >>>>>>>> [node05.cluster:21750] Type: Machine Number of child objects: 2 > >>>>>>>> Name=NULL > >>>>>>>> total=33080072KB > >>>>>>>> Backend=Linux > >>>>>>>> OSName=Linux > >>>>>>>> OSRelease=2.6.18-308.16.1.el5 > >>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >>>>>>>> Architecture=x86_64 > >>>>>>>> Cpuset: 0x000000ff > >>>>>>>> Online: 0x000000ff > >>>>>>>> Allowed: 0x000000ff > >>>>>>>> Bind CPU proc: TRUE > >>>>>>>> Bind CPU thread: TRUE > >>>>>>>> Bind MEM proc: FALSE > >>>>>>>> Bind MEM thread: TRUE > >>>>>>>> Type: NUMANode Number of child objects: 1 > >>>>>>>> Name=NULL > >>>>>>>> local=16532232KB > >>>>>>>> total=16532232KB > >>>>>>>> Cpuset: 0x0000000f > >>>>>>>> Online: 0x0000000f > >>>>>>>> Allowed: 0x0000000f > >>>>>>>> Type: Socket Number of child objects: 1 > >>>>>>>> > >>>>>>>> > >>>>>>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca > > ess_base_verbose > >>>> 5" > >>>>>> to > >>>>>>>> your cmd line and let's see what it thinks it found. > >>>>>>>>> > >>>>>>>>> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hi, I report one more problem with openmpi-1.7.4rc1, > >>>>>>>>>> which is more serious. > >>>>>>>>>> > >>>>>>>>>> For our 32 core nodes(AMD magny cours based) which has > >>>>>>>>>> 8 numa-nodes, "-bind-to numa" does not work. Without > >>>>>>>>>> this option, it works. For your infomation, at the > >>>>>>>>>> bottom of this mail, I added the lstopo information > >>>>>>>>>> of the node. > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Tetsuya Mishima > >>>>>>>>>> > >>>>>>>>>> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32 > >>>>>>>>>> qsub: waiting for job 8352.manage.cluster to start > >>>>>>>>>> qsub: job 8352.manage.cluster ready > >>>>>>>>>> > >>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings -bind-to > >>> numa > >>>>>>>> myprog > >>>>>>>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target NUMANode > >>>> type > >>>>>>>>>> Machine > >>>>>>>>>> > >>> > > -------------------------------------------------------------------------- > >>>>>>>>>> A request was made to bind to NUMA, but an appropriate target > >>> could > >>>>>> not > >>>>>>>>>> be found on node node03. > >>>>>>>>>> > >>> > > -------------------------------------------------------------------------- > >>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ > >>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog > >>>>>>>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8[hwt > >>> 0]]: > >>>>>>>>>> [./././././././.][B/././././././.][./././././././.][ > >>>>>>>>>> ./././././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 3 bound to socket 1[core 9[hwt > >>> 0]]: > >>>>>>>>>> [./././././././.][./B/./././././.][./././././././.][ > >>>>>>>>>> ./././././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16 [hwt > >>>> 0]]: > >>>>>>>>>> [./././././././.][./././././././.][B/././././././.] > >>>>>>>>>> [./././././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17 [hwt > >>>> 0]]: > >>>>>>>>>> [./././././././.][./././././././.][./B/./././././.] > >>>>>>>>>> [./././././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24 [hwt > >>>> 0]]: > >>>>>>>>>> [./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [B/././././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25 [hwt > >>>> 0]]: > >>>>>>>>>> [./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [./B/./././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0[hwt > >>> 0]]: > >>>>>>>>>> [B/././././././.][./././././././.][./././././././.][ > >>>>>>>>>> ./././././././.] > >>>>>>>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1[hwt > >>> 0]]: > >>>>>>>>>> [./B/./././././.][./././././././.][./././././././.][ > >>>>>>>>>> ./././././././.] > >>>>>>>>>> Hello world from process 2 of 8 > >>>>>>>>>> Hello world from process 5 of 8 > >>>>>>>>>> Hello world from process 4 of 8 > >>>>>>>>>> Hello world from process 3 of 8>>>>>>>>>> Hello world from process 1 of 8 > >>>>>>>>>> Hello world from process 7 of 8 > >>>>>>>>>> Hello world from process 6 of 8 > >>>>>>>>>> Hello world from process 0 of 8 > >>>>>>>>>> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics > >>>>>>>>>> Machine (126GB) > >>>>>>>>>> Socket L#0 (32GB) > >>>>>>>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) > >>>>>>>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + > >>> PU > >>>>>>>> L#0 > >>>>>>>>>> (P#0) > >>>>>>>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + > >>> PU > >>>>>>>> L#1 > >>>>>>>>>> (P#1) > >>>>>>>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + > >>> PU > >>>>>>>> L#2 > >>>>>>>>>> (P#2) > >>>>>>>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + > >>> PU > >>>>>>>> L#3 > >>>>>>>>>> (P#3) > >>>>>>>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) > >>>>>>>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + > >>> PU > >>>>>>>> L#4 > >>>>>>>>>> (P#4) > >>>>>>>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + > >>> PU > >>>>>>>> L#5 > >>>>>>>>>> (P#5) > >>>>>>>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + > >>> PU > >>>>>>>> L#6 > >>>>>>>>>> (P#6) > >>>>>>>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + > >>> PU>>>>>> L#7 > >>>>>>>>>> (P#7) > >>>>>>>>>> Socket L#1 (32GB) > >>>>>>>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) > >>>>>>>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + > >>> PU > >>>>>>>> L#8 > >>>>>>>>>> (P#8) > >>>>>>>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + > >>> PU > >>>>>>>> L#9 > >>>>>>>>>> (P#9) > >>>>>>>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core > >>> L#10 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#10 (P#10) > >>>>>>>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core > >>> L#11 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#11 (P#11) > >>>>>>>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) > >>>>>>>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core > >>> L#12 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#12 (P#12) > >>>>>>>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core > >>> L#13 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#13 (P#13) > >>>>>>>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core > >>> L#14 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#14 (P#14) > >>>>>>>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core > >>> L#15 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#15 (P#15) > >>>>>>>>>> Socket L#2 (32GB) > >>>>>>>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) > >>>>>>>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core > >>> L#16 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#16 (P#16) > >>>>>>>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core > >>> L#17 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + L1i > >>> L#18 (64KB) + Core L#18 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#18 (P#18) > >>>>>>>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core > >>> L#19 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#19 (P#19) > >>>>>>>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) > >>>>>>>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core > >>> L#20 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#20 (P#20) > >>>>>>>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core > >>> L#21 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#21 (P#21) > >>>>>>>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core > >>> L#22 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#22 (P#22) > >>>>>>>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core > >>> L#23 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#23 (P#23) > >>>>>>>>>> Socket L#3 (32GB) > >>>>>>>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) > >>>>>>>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core > >>> L#24 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 > >>>> (64KB) + Core L#25 + > >>>>>>>> PU > >>>>>>>>>> L#25 (P#25) > >>>>>>>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core > >>> L#26 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#26 (P#26) > >>>>>>>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core > >>> L#27 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#27 (P#27) > >>>>>>>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) > >>>>>>>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core > >>> L#28 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#28 (P#28) > >>>>>>>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core > >>> L#29 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#29 (P#29) > >>>>>>>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core > >>> L#30 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#30 (P#30) > >>>>>>>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core > >>> L#31 > >>>> + > >>>>>>>> PU > >>>>>>>>>> L#31 (P#31) > >>>>>>>>>> HostBridge L#0 > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>> Net L#0 "eth0" > >>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>> Net L#1 "eth1" > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>> Net L#2 "eth2" > >>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>> Net L#3 "eth3" > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCI 1000:0072 > >>>>>>>>>> Block L#4 "sdb" > >>>>>>>>>> Block L#5 "sda" > >>>>>>>>>> PCI 1002:4390 > >>>>>>>>>> Block L#6 "sr0" > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCI 102b:0532 > >>>>>>>>>> HostBridge L#7 > >>>>>>>>>> PCIBridge > >>>>>>>>>> PCI 15b3:6274 > >>>>>>>>>> Net L#7 "ib0" > >>>>>>>>>> OpenFabrics L#8 "mthca0" > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> users mailing list > >>>>>>>>>> us...@open-mpi.org > >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>> _______________________________________________ > >>>>>>>>> users mailing list > >>>>>>>>> us...@open-mpi.org>> > >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>> _______________________________________________ > >>>>>>>> users mailing list > >>>>>>>> us...@open-mpi.org > >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>> _______________________________________________ > >>>>>>> users mailing list > >>>>>>> us...@open-mpi.org > >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> us...@open-mpi.org > >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
