Hi Ralph,
Thank you very much. I tried many things such as: mpirun -np 2 -host node05 -report-bindings -mca rmaps lama -mca rmaps_lama_bind 1c myprog But every try failed. At least they were accepted by openmpi-1.7.3 as far as I remember. Anyway, please check it when you have a time, because using lama comes from my curiosity. Regards, Tetsuya Mishima > I'll try to take a look at it - my expectation is that lama might get stuck because you didn't tell it a pattern to map, and I doubt that code path has seen much testing. > > > On Dec 20, 2013, at 5:52 PM, tmish...@jcity.maeda.co.jp wrote: > > > > > > > Hi Ralph, I'm glad to hear that, thanks. > > > > By the way, yesterday I tried to check how lama in 1.7.4rc treat numa node. > > > > Then, even wiht this simple command line, it freezed without any massage: > > > > mpirun -np 2 -host node05 -mca rmaps lama myprog > > > > Could you check what happened? > > > > Is it better to open new thread or continue this thread? > > > > Regards, > > Tetsuya Mishima > > > > > >> I'll make it work so that NUMA can be either above or below socket > >> > >> On Dec 20, 2013, at 2:57 AM, tmish...@jcity.maeda.co.jp wrote: > >> > >>> > >>> > >>> Hi Brice, > >>> > >>> Thank you for your comment. I understand what you mean. > >>> > >>> My opinion was made just considering easy way to adjust the code for > >>> inversion of hierarchy in object tree. > >>> > >>> Tetsuya Mishima > >>> > >>> > >>>> I don't think there's any such difference. > >>>> Also, all these NUMA architectures are reported the same by hwloc, and > >>>> therefore used the same in Open MPI. > >>>> > >>>> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours > > (and > >>>> most recent AMD and Intel platforms). > >>>> > >>>> Brice > >>>> > >>>> > >>>> > >>>> Le 20/12/2013 11:33, tmish...@jcity.maeda.co.jp a écrit : > >>>>> > >>>>> Hi Ralph, > >>>>> > >>>>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache > >>>>> coherent)NUMA, > >>>>> which seems to be a little bit different from the traditional numa > >>> defined > >>>>> in openmpi. > >>>>> > >>>>> I notice that ccNUMA object is almost same as L3cache object. > >>>>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want > > to > >>> do. > >>>>> Therefore, "do not touch it" is one of the solution, I think ... > >>>>> > >>>>> Anyway, mixing up these two types of numa is the problem. > >>>>> > >>>>> Regards, > >>>>> Tetsuya Mishima > >>>>> > >>>>>> I can wait it'll be fixed in 1.7.5 or later, because putting > > "-bind-to > >>>>>> numa" > >>>>>> and "-map-by numa" at the same time works as a workaround. > >>>>>> > >>>>>> Thanks, > >>>>>> Tetsuya Mishima > >>>>>> > >>>>>>> Yeah, it will impact everything that uses hwloc topology maps, I > >>> fear. > >>>>>>> > >>>>>>> One side note: you'll need to add --hetero-nodes to your cmd line. > > If > >>>>> we > >>>>>> don't see that, we assume that all the node topologies are identical > > - > >>>>>> which clearly isn't true here. > >>>>>>> I'll try to resolve the hier inversion over the holiday - won't be > >>> for > >>>>>> 1.7.4, but hopefully for 1.7.5 > >>>>>>> Thanks > >>>>>>> Ralph > >>>>>>> > >>>>>>> On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> I think it's normal for AMD opteron having 8/16 cores such as > >>>>>>>> magny cours or interlagos. Because it usually has 2 numa nodes > >>>>>>>> in a cpu(socket), numa-node can not include a socket. This type > >>>>>>>> of hierarchy would be natural. > >>>>>>>> > >>>>>>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess) > >>>>>>>> > >>>>>>>> By the way, I think this inversion should affect rmaps_lama > > mapping. > >>>>>>>> > >>>>>>>> Tetsuya Mishima > >>>>>>>> > >>>>>>>>> Ick - yeah, that would be a problem. I haven't seen that type of > >>>>>>>> hierarchical inversion before - is node03 a different type of > > chip? > >>>>>>>>> Might take awhile for me to adjust the code to handle hier > >>>>>>>> inversion... :-( > >>>>>>>>> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hi Ralph, > >>>>>>>>>> > >>>>>>>>>> I found the reason. I attached the main part of output with 32 > >>>>>>>>>> core node(node03) and 8 core node(node05) at the bottom. > >>>>>>>>>> > >>>>>>>>>> From this information, socket of node03 includes numa-node. > >>>>>>>>>> On the other hand, numa-node of node05 includes socket. > >>>>>>>>>> The direction of object tree is opposite. > >>>>>>>>>> > >>>>>>>>>> Since "-map-by socket" may be assumed as default, > >>>>>>>>>> for node05, "-bind-to numa and -map-by socket" means > >>>>>>>>>> upward search. For node03, this should be downward. > >>>>>>>>>> > >>>>>>>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node > >>>>>>>>>> includes socket. Is it right? Then, upward search is assumed > >>>>>>>>>> in orte_rmaps_base_compute_bindings even for node03 when I > >>>>>>>>>> put "-bind-to numa and -map-by socket" option. > >>>>>>>>>> > >>>>>>>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage > >>>>>>>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job > >>>>> [38286,1] > >>>>>>>> with > >>>>>>>>>> policy NUMA > >>>>>>>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job [38286,1] > >>>>> with > >>>>>>>>>> bindings NUMA > >>>>>>>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> Machine > >>>>>>>>>> > >>>>>>>>>> That's the reason of this trouble. Therefore, adding "-map-by > >>> core" > >>>>>>>> works. > >>>>>>>>>> (mapping pattern seems to be strange ...) > >>>>>>>>>> > >>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by core > >>>>>>>>>> -report-bindings myprog > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>> Cache > >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > >>>>> type > >>>>>>>>>> NUMANode > >>>>>>>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>>>> > >>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>>>> > >>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>>>> > >>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>>>> > >>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>>>> > >>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>>>>>> cket 0[core 7[hwt 0]]: > >>>>>>>>>> > >>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>>>> > >>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt > >>> 0]], > >>>>>>>> socket > >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>>>>>> cket 0[core 3[hwt 0]]: > >>>>>>>>>> > >>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>>>>>> Hello world from process 6 of 8 > >>>>>>>>>> Hello world from process 5 of 8 > >>>>>>>>>> Hello world from process 0 of 8 > >>>>>>>>>> Hello world from process 7 of 8 > >>>>>>>>>> Hello world from process 3 of 8 > >>>>>>>>>> Hello world from process 4 of 8 > >>>>>>>>>> Hello world from process 2 of 8 > >>>>>>>>>> Hello world from process 1 of 8 > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Tetsuya Mishima > >>>>>>>>>> > >>>>>>>>>> [node03.cluster:15508] Type: Machine Number of child objects: 4 > >>>>>>>>>> Name=NULL > >>>>>>>>>> total=132358820KB > >>>>>>>>>> Backend=Linux > >>>>>>>>>> OSName=Linux > >>>>>>>>>> OSRelease=2.6.18-308.16.1.el5 > >>>>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >>>>>>>>>> Architecture=x86_64 > >>>>>>>>>> Cpuset: 0xffffffff > >>>>>>>>>> Online: 0xffffffff > >>>>>>>>>> Allowed: 0xffffffff > >>>>>>>>>> Bind CPU proc: TRUE > >>>>>>>>>> Bind CPU thread: TRUE > >>>>>>>>>> Bind MEM proc: FALSE > >>>>>>>>>> Bind MEM thread: TRUE > >>>>>>>>>> Type: Socket Number of child objects: 2 > >>>>>>>>>> Name=NULL > >>>>>>>>>> total=33071780KB > >>>>>>>>>> CPUModel="AMD Opteron(tm) Processor 6136" > >>>>>>>>>> Cpuset: 0x000000ff > >>>>>>>>>> Online: 0x000000ff > >>>>>>>>>> Allowed: 0x000000ff > >>>>>>>>>> Type: NUMANode Number of child objects: 1 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> [node05.cluster:21750] Type: Machine Number of child objects: 2 > >>>>>>>>>> Name=NULL > >>>>>>>>>> total=33080072KB > >>>>>>>>>> Backend=Linux > >>>>>>>>>> OSName=Linux > >>>>>>>>>> OSRelease=2.6.18-308.16.1.el5 > >>>>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >>>>>>>>>> Architecture=x86_64 > >>>>>>>>>> Cpuset: 0x000000ff > >>>>>>>>>> Online: 0x000000ff > >>>>>>>>>> Allowed: 0x000000ff > >>>>>>>>>> Bind CPU proc: TRUE > >>>>>>>>>> Bind CPU thread: TRUE > >>>>>>>>>> Bind MEM proc: FALSE > >>>>>>>>>> Bind MEM thread: TRUE > >>>>>>>>>> Type: NUMANode Number of child objects: 1 > >>>>>>>>>> Name=NULL > >>>>>>>>>> local=16532232KB > >>>>>>>>>> total=16532232KB > >>>>>>>>>> Cpuset: 0x0000000f > >>>>>>>>>> Online: 0x0000000f > >>>>>>>>>> Allowed: 0x0000000f > >>>>>>>>>> Type: Socket Number of child objects: 1 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca > >>> ess_base_verbose > >>>>>> 5" > >>>>>>>> to > >>>>>>>>>> your cmd line and let's see what it thinks it found. > >>>>>>>>>>> > >>>>>>>>>>> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Hi, I report one more problem with openmpi-1.7.4rc1, > >>>>>>>>>>>> which is more serious. > >>>>>>>>>>>> > >>>>>>>>>>>> For our 32 core nodes(AMD magny cours based) which has > >>>>>>>>>>>> 8 numa-nodes, "-bind-to numa" does not work. Without > >>>>>>>>>>>> this option, it works. For your infomation, at the > >>>>>>>>>>>> bottom of this mail, I added the lstopo information > >>>>>>>>>>>> of the node. > >>>>>>>>>>>> > >>>>>>>>>>>> Regards, > >>>>>>>>>>>> Tetsuya Mishima > >>>>>>>>>>>> > >>>>>>>>>>>> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32 > >>>>>>>>>>>> qsub: waiting for job 8352.manage.cluster to start > >>>>>>>>>>>> qsub: job 8352.manage.cluster ready > >>>>>>>>>>>> > >>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings -bind-to > >>>>> numa > >>>>>>>>>> myprog > >>>>>>>>>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target > > NUMANode > >>>>>> type > >>>>>>>>>>>> Machine > >>>>>>>>>>>> > >>>>> > >>> > > -------------------------------------------------------------------------- > >>>>>>>>>>>> A request was made to bind to NUMA, but an appropriate target > >>>>> could > >>>>>>>> not > >>>>>>>>>>>> be found on node node03. > >>>>>>>>>>>> > >>>>> > >>> > > -------------------------------------------------------------------------- > >>>>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ > >>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8 [hwt > >>>>> 0]]: > >>>>>>>>>>>> [./././././././.][B/././././././.][./././././././.][ > >>>>>>>>>>>> ./././././././.]>>>>>>>>>>>> [node03.cluster:15282] MCW rank 3 bound to socket 1[core 9[hwt > >>>>> 0]]: > >>>>>>>>>>>> [./././././././.][./B/./././././.][./././././././.][ > >>>>>>>>>>>> ./././././././.] > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16 > > [hwt > >>>>>> 0]]: > >>>>>>>>>>>> [./././././././.][./././././././.][B/././././././.] > >>>>>>>>>>>> [./././././././.] > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17 > > [hwt > >>>>>> 0]]: > >>>>>>>>>>>> [./././././././.][./././././././.][./B/./././././.] > >>>>>>>>>>>> [./././././././.] > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24 > > [hwt > >>>>>> 0]]: > >>>>>>>>>>>> [./././././././.][./././././././.][./././././././.] > >>>>>>>>>>>> [B/././././././.] > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25 > > [hwt > >>>>>> 0]]: > >>>>>>>>>>>> [./././././././.][./././././././.][./././././././.] > >>>>>>>>>>>> [./B/./././././.] > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0 [hwt > >>>>> 0]]: > >>>>>>>>>>>> [B/././././././.][./././././././.][./././././././.][ > >>>>>>>>>>>> ./././././././.] > >>>>>>>>>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1 [hwt > >>>>> 0]]: > >>>>>>>>>>>> [./B/./././././.][./././././././.][./././././././.][ > >>>>>>>>>>>> ./././././././.] > >>>>>>>>>>>> Hello world from process 2 of 8 > >>>>>>>>>>>> Hello world from process 5 of 8 > >>>>>>>>>>>> Hello world from process 4 of 8 > >>>>>>>>>>>> Hello world from process 3 of 8>>>>>>>>>> Hello world from > > process 1 of 8 > >>>>>>>>>>>> Hello world from process 7 of 8 > >>>>>>>>>>>> Hello world from process 6 of 8 > >>>>>>>>>>>> Hello world from process 0 of 8 > >>>>>>>>>>>> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics > >>>>>>>>>>>> Machine (126GB) > >>>>>>>>>>>> Socket L#0 (32GB) > >>>>>>>>>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) > >>>>>>>>>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 > > + > >>>>> PU > >>>>>>>>>> L#0 > >>>>>>>>>>>> (P#0) > >>>>>>>>>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 > > + > >>>>> PU > >>>>>>>>>> L#1 > >>>>>>>>>>>> (P#1) > >>>>>>>>>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 > > + > >>>>> PU > >>>>>>>>>> L#2 > >>>>>>>>>>>> (P#2) > >>>>>>>>>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 > > + > >>>>> PU > >>>>>>>>>> L#3 > >>>>>>>>>>>> (P#3) > >>>>>>>>>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) > >>>>>>>>>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 > > + > >>>>> PU > >>>>>>>>>> L#4 > >>>>>>>>>>>> (P#4) > >>>>>>>>>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 > > + > >>>>> PU > >>>>>>>>>> L#5 > >>>>>>>>>>>> (P#5) > >>>>>>>>>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 > > + > >>>>> PU > >>>>>>>>>> L#6 > >>>>>>>>>>>> (P#6) > >>>>>>>>>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 > > + > >>>>> PU>>>>>> L#7 > >>>>>>>>>>>> (P#7) > >>>>>>>>>>>> Socket L#1 (32GB) > >>>>>>>>>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) > >>>>>>>>>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 > > + > >>>>> PU > >>>>>>>>>> L#8 > >>>>>>>>>>>> (P#8) > >>>>>>>>>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 > > + > >>>>> PU > >>>>>>>>>> L#9 > >>>>>>>>>>>> (P#9) > >>>>>>>>>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core > >>>>> L#10 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#10 (P#10) > >>>>>>>>>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core > >>>>> L#11 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#11 (P#11) > >>>>>>>>>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) > >>>>>>>>>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core > >>>>> L#12 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#12 (P#12) > >>>>>>>>>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core > >>>>> L#13 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#13 (P#13) > >>>>>>>>>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core > >>>>> L#14 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#14 (P#14) > >>>>>>>>>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core > >>>>> L#15 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#15 (P#15) > >>>>>>>>>>>> Socket L#2 (32GB) > >>>>>>>>>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) > >>>>>>>>>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core > >>>>> L#16 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#16 (P#16) > >>>>>>>>>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core > >>>>> L#17 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + L1i > >>>>> L#18 (64KB) + Core L#18 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#18 (P#18) > >>>>>>>>>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core > >>>>> L#19 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#19 (P#19) > >>>>>>>>>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) > >>>>>>>>>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core > >>>>> L#20 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#20 (P#20) > >>>>>>>>>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core > >>>>> L#21 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#21 (P#21) > >>>>>>>>>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core > >>>>> L#22 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#22 (P#22) > >>>>>>>>>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core > >>>>> L#23 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#23 (P#23) > >>>>>>>>>>>> Socket L#3 (32GB) > >>>>>>>>>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) > >>>>>>>>>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core > >>>>> L#24 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i > > L#25 > >>>>>> (64KB) + Core L#25 + > >>>>>>>>>> PU > >>>>>>>>>>>> L#25 (P#25) > >>>>>>>>>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core > >>>>> L#26 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#26 (P#26) > >>>>>>>>>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core > >>>>> L#27 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#27 (P#27) > >>>>>>>>>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) > >>>>>>>>>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core > >>>>> L#28 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#28 (P#28) > >>>>>>>>>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core > >>>>> L#29 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#29 (P#29) > >>>>>>>>>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core > >>>>> L#30 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#30 (P#30) > >>>>>>>>>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core > >>>>> L#31 > >>>>>> + > >>>>>>>>>> PU > >>>>>>>>>>>> L#31 (P#31) > >>>>>>>>>>>> HostBridge L#0 > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>>>> Net L#0 "eth0" > >>>>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>>>> Net L#1 "eth1" > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>>>> Net L#2 "eth2" > >>>>>>>>>>>> PCI 14e4:1639 > >>>>>>>>>>>> Net L#3 "eth3" > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCI 1000:0072 > >>>>>>>>>>>> Block L#4 "sdb" > >>>>>>>>>>>> Block L#5 "sda" > >>>>>>>>>>>> PCI 1002:4390 > >>>>>>>>>>>> Block L#6 "sr0" > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCI 102b:0532 > >>>>>>>>>>>> HostBridge L#7 > >>>>>>>>>>>> PCIBridge > >>>>>>>>>>>> PCI 15b3:6274 > >>>>>>>>>>>> Net L#7 "ib0" > >>>>>>>>>>>> OpenFabrics L#8 "mthca0" > >>>>>>>>>>>> > >>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>> users mailing list > >>>>>>>>>>>> us...@open-mpi.org > >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> users mailing list > >>>>>>>>>>> us...@open-mpi.org>> > >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> users mailing list > >>>>>>>>>> us...@open-mpi.org > >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>>> _______________________________________________ > >>>>>>>>> users mailing list > >>>>>>>>> us...@open-mpi.org > >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>>> _______________________________________________ > >>>>>>>> users mailing list > >>>>>>>> us...@open-mpi.org > >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>> _______________________________________________ > >>>>>>> users mailing list > >>>>>>> us...@open-mpi.org > >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> us...@open-mpi.org > >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>> > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
