Hi, I'd like to revive this thread, since I am still periodically getting errors of this type. I have built 1.8.1 with --enable-debug and run with -mca btl_openib_verbose 10. Unfortunately, this doesn't seem to provide any additional information that I can find useful. I've gone ahead and attached a dump of the output under 1.8.1. The key lines are:
-------------------------------------------------------------------------- Open MPI detected two different OpenFabrics transport types in the same Infiniband network. Such mixed network trasport configuration is not supported by Open MPI. Local host: w1 Local adapter: mlx4_0 (vendor 0x2c9, part ID 26428) Local transport type: MCA_BTL_OPENIB_TRANSPORT_IB Remote host: w16 Remote Adapter: (vendor 0x2c9, part ID 26428) Remote transport type: MCA_BTL_OPENIB_TRANSPORT_UNKNOWN ------------------------------------------------------------------------- Note that the vendor and part IDs are the same. If I immediately run on the same two nodes using MVAPICH2, everything is fine. I'm really very befuddled by this. OpenMPI sees that the two cards are the same and made by the same vendor, yet it thinks the transport types are different (and one is unknown). I'm hoping someone with some experience with how the OpenIB BTL works can shed some light on this problem... Tim On Fri, May 9, 2014 at 7:39 PM, Joshua Ladd <jladd.m...@gmail.com> wrote: > > Just wondering if you've tried with the latest stable OMPI, 1.8.1? I'm > wondering if this is an issue with the OOB. If you have a debug build, you > can run -mca btl_openib_verbose 10 > > Josh > > > On Fri, May 9, 2014 at 6:26 PM, Joshua Ladd <jladd.m...@gmail.com> wrote: > >> Hi, Tim >> >> Run "ibstat" on each host: >> >> 1. Make sure the adapters are alive and active. >> >> 2. Look at the Link Layer settings for host w34. Does it match host w4's? >> >> >> Josh >> >> >> On Fri, May 9, 2014 at 1:18 PM, Tim Miller <btamil...@gmail.com> wrote: >> >>> Hi All, >>> >>> We're using OpenMPI 1.7.3 with Mellanox ConnectX InfiniBand adapters, >>> and periodically our jobs abort at start-up with the following error: >>> >>> === >>> Open MPI detected two different OpenFabrics transport types in the same >>> Infiniband network. >>> Such mixed network trasport configuration is not supported by Open MPI. >>> >>> Local host: w4 >>> Local adapter: mlx4_0 (vendor 0x2c9, part ID 26428) >>> Local transport type: MCA_BTL_OPENIB_TRANSPORT_IB >>> >>> Remote host: w34 >>> Remote Adapter: (vendor 0x2c9, part ID 26428) >>> Remote transport type: MCA_BTL_OPENIB_TRANSPORT_UNKNOWN >>> === >>> >>> I've done a bit of googling and not found very much. We do not see this >>> issue when we run with MVAPICH2 on the same sets of nodes. >>> >>> Any advice or thoughts would be very welcome, as I am stumped by what >>> causes this. The nodes are all running Scientific Linux 6 with Mellanox >>> drivers installed via the SL-provided RPMs. >>> >>> Tim >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
[w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],12][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],12][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],12][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],12][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],13][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],13][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],13][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],13][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],14][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],14][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],14][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],14][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],16][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],16][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],16][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],16][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],18][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],18][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],18][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],18][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],20][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],20][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],20][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],20][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],15][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],15][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],15][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],15][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],17][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],17][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],17][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],17][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],22][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],22][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],22][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],22][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],19][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],19][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],19][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],19][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],21][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],21][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],21][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],21][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w16][[56731,1],23][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w16][[56731,1],23][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w16][[56731,1],23][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0 [w16][[56731,1],23][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: default [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],8][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],8][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],10][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],10][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],11][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],11][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],3][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],3][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],4][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],4][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],7][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],7][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],5][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],5][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],6][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],6][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],9][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],9][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],1][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],1][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],0][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],0][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon [w1][[56731,1],2][btl_openib_ini.c:170:ompi_btl_openib_ini_query] Querying INI files for vendor 0x02c9, part ID 26428 [w1][[56731,1],2][btl_openib_ini.c:189:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4279] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4281] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4283] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4282] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4284] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! [w1.lobos.nih.gov:4278] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! [w1.lobos.nih.gov:4277] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4280] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! [w1.lobos.nih.gov:4276] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4274] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4275] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w1.lobos.nih.gov:4273] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! -------------------------------------------------------------------------- Open MPI detected two different OpenFabrics transport types in the same Infiniband network. Such mixed network trasport configuration is not supported by Open MPI. Local host: w1 Local adapter: mlx4_0 (vendor 0x2c9, part ID 26428) Local transport type: MCA_BTL_OPENIB_TRANSPORT_IB Remote host: w16 Remote Adapter: (vendor 0x2c9, part ID 26428) Remote transport type: MCA_BTL_OPENIB_TRANSPORT_UNKNOWN -------------------------------------------------------------------------- -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[56731,1],6]) is on host: w1 Process 2 ([[56731,1],12]) is on host: w1 BTLs attempted: openib self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6320] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! -------------------------------------------------------------------------- MPI_INIT has failed because at least one MPI process is unreachable from another. This *usually* means that an underlying communication plugin -- such as a BTL or an MTL -- has either not loaded or not allowed itself to be used. Your MPI job will now abort. You may wish to try to narrow down the problem; * Check the output of ompi_info to see which BTL/MTL plugins are available. * Run your application with MPI_THREAD_SINGLE. * Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose, if using MTL-based communications) to see exactly which communication plugins were considered and/or discarded. -------------------------------------------------------------------------- *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6324] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6326] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6322] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6323] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6325] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6327] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6319] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6321] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6316] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6317] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [w16.lobos.nih.gov:6318] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed! ------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. ------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[56731,1],1] Exit code: 1 --------------------------------------------------------------------------
