I'll make it work so that NUMA can be either above or below socket On Dec 20, 2013, at 2:57 AM, tmish...@jcity.maeda.co.jp wrote:
> > > Hi Brice, > > Thank you for your comment. I understand what you mean. > > My opinion was made just considering easy way to adjust the code for > inversion of hierarchy in object tree. > > Tetsuya Mishima > > >> I don't think there's any such difference. >> Also, all these NUMA architectures are reported the same by hwloc, and >> therefore used the same in Open MPI. >> >> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours (and >> most recent AMD and Intel platforms). >> >> Brice >> >> >> >> Le 20/12/2013 11:33, tmish...@jcity.maeda.co.jp a écrit : >>> >>> Hi Ralph, >>> >>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache >>> coherent)NUMA, >>> which seems to be a little bit different from the traditional numa > defined >>> in openmpi. >>> >>> I notice that ccNUMA object is almost same as L3cache object. >>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want to > do. >>> Therefore, "do not touch it" is one of the solution, I think ... >>> >>> Anyway, mixing up these two types of numa is the problem. >>> >>> Regards, >>> Tetsuya Mishima >>> >>>> I can wait it'll be fixed in 1.7.5 or later, because putting "-bind-to >>>> numa" >>>> and "-map-by numa" at the same time works as a workaround. >>>> >>>> Thanks, >>>> Tetsuya Mishima >>>> >>>>> Yeah, it will impact everything that uses hwloc topology maps, I > fear. >>>>> >>>>> One side note: you'll need to add --hetero-nodes to your cmd line. If >>> we >>>> don't see that, we assume that all the node topologies are identical - >>>> which clearly isn't true here. >>>>> I'll try to resolve the hier inversion over the holiday - won't be > for >>>> 1.7.4, but hopefully for 1.7.5 >>>>> Thanks >>>>> Ralph >>>>> >>>>> On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: >>>>> >>>>>> >>>>>> I think it's normal for AMD opteron having 8/16 cores such as >>>>>> magny cours or interlagos. Because it usually has 2 numa nodes >>>>>> in a cpu(socket), numa-node can not include a socket. This type >>>>>> of hierarchy would be natural. >>>>>> >>>>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess) >>>>>> >>>>>> By the way, I think this inversion should affect rmaps_lama mapping. >>>>>> >>>>>> Tetsuya Mishima >>>>>> >>>>>>> Ick - yeah, that would be a problem. I haven't seen that type of >>>>>> hierarchical inversion before - is node03 a different type of chip? >>>>>>> Might take awhile for me to adjust the code to handle hier >>>>>> inversion... :-( >>>>>>> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi Ralph, >>>>>>>> >>>>>>>> I found the reason. I attached the main part of output with 32 >>>>>>>> core node(node03) and 8 core node(node05) at the bottom. >>>>>>>> >>>>>>>> From this information, socket of node03 includes numa-node. >>>>>>>> On the other hand, numa-node of node05 includes socket. >>>>>>>> The direction of object tree is opposite. >>>>>>>> >>>>>>>> Since "-map-by socket" may be assumed as default, >>>>>>>> for node05, "-bind-to numa and -map-by socket" means >>>>>>>> upward search. For node03, this should be downward. >>>>>>>> >>>>>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node >>>>>>>> includes socket. Is it right? Then, upward search is assumed >>>>>>>> in orte_rmaps_base_compute_bindings even for node03 when I >>>>>>>> put "-bind-to numa and -map-by socket" option. >>>>>>>> >>>>>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage >>>>>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job >>> [38286,1] >>>>>> with >>>>>>>> policy NUMA >>>>>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job [38286,1] >>> with >>>>>>>> bindings NUMA >>>>>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target NUMANode >>> type >>>>>>>> Machine >>>>>>>> >>>>>>>> That's the reason of this trouble. Therefore, adding "-map-by > core" >>>>>> works. >>>>>>>> (mapping pattern seems to be strange ...) >>>>>>>> >>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by core >>>>>>>> -report-bindings myprog >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>> Cache >>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode >>> type >>>>>>>> NUMANode >>>>>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>>>>>> cket 0[core 3[hwt 0]]: >>>>>>>> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>>>>>> cket 0[core 3[hwt 0]]: >>>>>>>> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>>>>>> cket 0[core 7[hwt 0]]: >>>>>>>> >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>>>>>> cket 0[core 7[hwt 0]]: >>>>>>>> >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>>>>>> cket 0[core 7[hwt 0]]: >>>>>>>> >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so >>>>>>>> cket 0[core 7[hwt 0]]: >>>>>>>> >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>>>>>> cket 0[core 3[hwt 0]]: >>>>>>>> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>>>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt > 0]], >>>>>> socket >>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so >>>>>>>> cket 0[core 3[hwt 0]]: >>>>>>>> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] >>>>>>>> Hello world from process 6 of 8 >>>>>>>> Hello world from process 5 of 8 >>>>>>>> Hello world from process 0 of 8 >>>>>>>> Hello world from process 7 of 8 >>>>>>>> Hello world from process 3 of 8 >>>>>>>> Hello world from process 4 of 8 >>>>>>>> Hello world from process 2 of 8 >>>>>>>> Hello world from process 1 of 8 >>>>>>>> >>>>>>>> Regards, >>>>>>>> Tetsuya Mishima >>>>>>>> >>>>>>>> [node03.cluster:15508] Type: Machine Number of child objects: 4 >>>>>>>> Name=NULL >>>>>>>> total=132358820KB >>>>>>>> Backend=Linux >>>>>>>> OSName=Linux >>>>>>>> OSRelease=2.6.18-308.16.1.el5 >>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" >>>>>>>> Architecture=x86_64 >>>>>>>> Cpuset: 0xffffffff >>>>>>>> Online: 0xffffffff >>>>>>>> Allowed: 0xffffffff >>>>>>>> Bind CPU proc: TRUE >>>>>>>> Bind CPU thread: TRUE >>>>>>>> Bind MEM proc: FALSE >>>>>>>> Bind MEM thread: TRUE >>>>>>>> Type: Socket Number of child objects: 2 >>>>>>>> Name=NULL >>>>>>>> total=33071780KB >>>>>>>> CPUModel="AMD Opteron(tm) Processor 6136" >>>>>>>> Cpuset: 0x000000ff >>>>>>>> Online: 0x000000ff >>>>>>>> Allowed: 0x000000ff >>>>>>>> Type: NUMANode Number of child objects: 1 >>>>>>>> >>>>>>>> >>>>>>>> [node05.cluster:21750] Type: Machine Number of child objects: 2 >>>>>>>> Name=NULL >>>>>>>> total=33080072KB >>>>>>>> Backend=Linux >>>>>>>> OSName=Linux >>>>>>>> OSRelease=2.6.18-308.16.1.el5 >>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" >>>>>>>> Architecture=x86_64 >>>>>>>> Cpuset: 0x000000ff >>>>>>>> Online: 0x000000ff >>>>>>>> Allowed: 0x000000ff >>>>>>>> Bind CPU proc: TRUE >>>>>>>> Bind CPU thread: TRUE >>>>>>>> Bind MEM proc: FALSE >>>>>>>> Bind MEM thread: TRUE >>>>>>>> Type: NUMANode Number of child objects: 1 >>>>>>>> Name=NULL >>>>>>>> local=16532232KB >>>>>>>> total=16532232KB >>>>>>>> Cpuset: 0x0000000f >>>>>>>> Online: 0x0000000f >>>>>>>> Allowed: 0x0000000f >>>>>>>> Type: Socket Number of child objects: 1 >>>>>>>> >>>>>>>> >>>>>>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca > ess_base_verbose >>>> 5" >>>>>> to >>>>>>>> your cmd line and let's see what it thinks it found. >>>>>>>>> >>>>>>>>> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, I report one more problem with openmpi-1.7.4rc1, >>>>>>>>>> which is more serious. >>>>>>>>>> >>>>>>>>>> For our 32 core nodes(AMD magny cours based) which has >>>>>>>>>> 8 numa-nodes, "-bind-to numa" does not work. Without >>>>>>>>>> this option, it works. For your infomation, at the >>>>>>>>>> bottom of this mail, I added the lstopo information >>>>>>>>>> of the node. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Tetsuya Mishima >>>>>>>>>> >>>>>>>>>> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32 >>>>>>>>>> qsub: waiting for job 8352.manage.cluster to start >>>>>>>>>> qsub: job 8352.manage.cluster ready >>>>>>>>>> >>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings -bind-to >>> numa >>>>>>>> myprog >>>>>>>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target NUMANode >>>> type >>>>>>>>>> Machine >>>>>>>>>> >>> > -------------------------------------------------------------------------- >>>>>>>>>> A request was made to bind to NUMA, but an appropriate target >>> could >>>>>> not >>>>>>>>>> be found on node node03. >>>>>>>>>> >>> > -------------------------------------------------------------------------- >>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ >>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog >>>>>>>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8[hwt >>> 0]]: >>>>>>>>>> [./././././././.][B/././././././.][./././././././.][ >>>>>>>>>> ./././././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 3 bound to socket 1[core 9[hwt >>> 0]]: >>>>>>>>>> [./././././././.][./B/./././././.][./././././././.][ >>>>>>>>>> ./././././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16[hwt >>>> 0]]: >>>>>>>>>> [./././././././.][./././././././.][B/././././././.] >>>>>>>>>> [./././././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17[hwt >>>> 0]]: >>>>>>>>>> [./././././././.][./././././././.][./B/./././././.] >>>>>>>>>> [./././././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24[hwt >>>> 0]]: >>>>>>>>>> [./././././././.][./././././././.][./././././././.] >>>>>>>>>> [B/././././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25[hwt >>>> 0]]: >>>>>>>>>> [./././././././.][./././././././.][./././././././.] >>>>>>>>>> [./B/./././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0[hwt >>> 0]]: >>>>>>>>>> [B/././././././.][./././././././.][./././././././.][ >>>>>>>>>> ./././././././.] >>>>>>>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1[hwt >>> 0]]: >>>>>>>>>> [./B/./././././.][./././././././.][./././././././.][ >>>>>>>>>> ./././././././.] >>>>>>>>>> Hello world from process 2 of 8 >>>>>>>>>> Hello world from process 5 of 8 >>>>>>>>>> Hello world from process 4 of 8 >>>>>>>>>> Hello world from process 3 of 8 >>>>>>>>>> Hello world from process 1 of 8 >>>>>>>>>> Hello world from process 7 of 8 >>>>>>>>>> Hello world from process 6 of 8 >>>>>>>>>> Hello world from process 0 of 8 >>>>>>>>>> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics >>>>>>>>>> Machine (126GB) >>>>>>>>>> Socket L#0 (32GB) >>>>>>>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) >>>>>>>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + >>> PU >>>>>>>> L#0 >>>>>>>>>> (P#0) >>>>>>>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + >>> PU >>>>>>>> L#1 >>>>>>>>>> (P#1) >>>>>>>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + >>> PU >>>>>>>> L#2 >>>>>>>>>> (P#2) >>>>>>>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + >>> PU >>>>>>>> L#3 >>>>>>>>>> (P#3) >>>>>>>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) >>>>>>>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + >>> PU >>>>>>>> L#4 >>>>>>>>>> (P#4) >>>>>>>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + >>> PU >>>>>>>> L#5 >>>>>>>>>> (P#5) >>>>>>>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + >>> PU >>>>>>>> L#6 >>>>>>>>>> (P#6) >>>>>>>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + >>> PU>>>>>> L#7 >>>>>>>>>> (P#7) >>>>>>>>>> Socket L#1 (32GB) >>>>>>>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) >>>>>>>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + >>> PU >>>>>>>> L#8 >>>>>>>>>> (P#8) >>>>>>>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + >>> PU >>>>>>>> L#9 >>>>>>>>>> (P#9) >>>>>>>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core >>> L#10 >>>> + >>>>>>>> PU >>>>>>>>>> L#10 (P#10) >>>>>>>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core >>> L#11 >>>> + >>>>>>>> PU >>>>>>>>>> L#11 (P#11) >>>>>>>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) >>>>>>>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core >>> L#12 >>>> + >>>>>>>> PU >>>>>>>>>> L#12 (P#12) >>>>>>>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core >>> L#13 >>>> + >>>>>>>> PU >>>>>>>>>> L#13 (P#13) >>>>>>>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core >>> L#14 >>>> + >>>>>>>> PU >>>>>>>>>> L#14 (P#14) >>>>>>>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core >>> L#15 >>>> + >>>>>>>> PU >>>>>>>>>> L#15 (P#15) >>>>>>>>>> Socket L#2 (32GB) >>>>>>>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) >>>>>>>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core >>> L#16 >>>> + >>>>>>>> PU >>>>>>>>>> L#16 (P#16) >>>>>>>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core >>> L#17 >>>> + >>>>>>>> PU >>>>>>>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + L1i >>> L#18 (64KB) + Core L#18 >>>> + >>>>>>>> PU >>>>>>>>>> L#18 (P#18) >>>>>>>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core >>> L#19 >>>> + >>>>>>>> PU >>>>>>>>>> L#19 (P#19) >>>>>>>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) >>>>>>>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core >>> L#20 >>>> + >>>>>>>> PU >>>>>>>>>> L#20 (P#20) >>>>>>>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core >>> L#21 >>>> + >>>>>>>> PU >>>>>>>>>> L#21 (P#21) >>>>>>>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core >>> L#22 >>>> + >>>>>>>> PU >>>>>>>>>> L#22 (P#22) >>>>>>>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core >>> L#23 >>>> + >>>>>>>> PU >>>>>>>>>> L#23 (P#23) >>>>>>>>>> Socket L#3 (32GB) >>>>>>>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) >>>>>>>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core >>> L#24 >>>> + >>>>>>>> PU >>>>>>>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 >>>> (64KB) + Core L#25 + >>>>>>>> PU >>>>>>>>>> L#25 (P#25) >>>>>>>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core >>> L#26 >>>> + >>>>>>>> PU >>>>>>>>>> L#26 (P#26) >>>>>>>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core >>> L#27 >>>> + >>>>>>>> PU >>>>>>>>>> L#27 (P#27) >>>>>>>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) >>>>>>>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core >>> L#28 >>>> + >>>>>>>> PU >>>>>>>>>> L#28 (P#28) >>>>>>>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core >>> L#29 >>>> + >>>>>>>> PU >>>>>>>>>> L#29 (P#29) >>>>>>>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core >>> L#30 >>>> + >>>>>>>> PU >>>>>>>>>> L#30 (P#30) >>>>>>>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core >>> L#31 >>>> + >>>>>>>> PU >>>>>>>>>> L#31 (P#31) >>>>>>>>>> HostBridge L#0 >>>>>>>>>> PCIBridge >>>>>>>>>> PCI 14e4:1639 >>>>>>>>>> Net L#0 "eth0" >>>>>>>>>> PCI 14e4:1639 >>>>>>>>>> Net L#1 "eth1" >>>>>>>>>> PCIBridge >>>>>>>>>> PCI 14e4:1639 >>>>>>>>>> Net L#2 "eth2" >>>>>>>>>> PCI 14e4:1639 >>>>>>>>>> Net L#3 "eth3" >>>>>>>>>> PCIBridge >>>>>>>>>> PCIBridge >>>>>>>>>> PCIBridge >>>>>>>>>> PCI 1000:0072 >>>>>>>>>> Block L#4 "sdb" >>>>>>>>>> Block L#5 "sda" >>>>>>>>>> PCI 1002:4390 >>>>>>>>>> Block L#6 "sr0" >>>>>>>>>> PCIBridge >>>>>>>>>> PCI 102b:0532 >>>>>>>>>> HostBridge L#7 >>>>>>>>>> PCIBridge >>>>>>>>>> PCI 15b3:6274 >>>>>>>>>> Net L#7 "ib0" >>>>>>>>>> OpenFabrics L#8 "mthca0" >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org>> >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
