> On 25 Mar 2015, at 16:52 , Ralph Castain <r...@open-mpi.org> wrote: > > Hmmm…okay, sorry to keep drilling down here, but let’s try adding “-mca > sec_base_verbose 100” now
> /u/sciteam/marksant/openmpi/installation/bin/mpirun -mca oob_base_verbose 100 > -mca sec_base_verbose 100 ./a.out [nid25257:09727] mca: base: components_register: registering sec components [nid25257:09727] mca: base: components_register: found loaded component munge [nid25257:09727] mca: base: components_register: component munge has no register or open function [nid25257:09727] mca: base: components_register: found loaded component basic [nid25257:09727] mca: base: components_register: component basic has no register or open function [nid25257:09727] mca: base: components_open: opening sec components [nid25257:09727] mca: base: components_open: found loaded component munge [nid25257:09727] mca: base: components_open: component munge open function successful [nid25257:09727] mca: base: components_open: found loaded component basic [nid25257:09727] mca: base: components_open: component basic open function successful [nid25257:09727] mca:sec:select: checking available component munge [nid25257:09727] mca:sec:select: Querying component [munge] [nid25257:09727] sec: munge init [nid25257:09727] mca:sec:select: checking available component basic [nid25257:09727] mca:sec:select: Querying component [basic] [nid25257:09727] mca: base: components_register: registering oob components [nid25257:09727] mca: base: components_register: found loaded component usock [nid25257:09727] mca: base: components_register: component usock register function successful [nid25257:09727] mca: base: components_register: found loaded component alps [nid25257:09727] mca: base: components_register: component alps register function successful [nid25257:09727] mca: base: components_register: found loaded component ud [nid25257:09727] mca: base: components_register: component ud register function successful [nid25257:09727] mca: base: components_register: found loaded component tcp [nid25257:09727] mca: base: components_register: component tcp register function successful [nid25257:09727] mca: base: components_open: opening oob components [nid25257:09727] mca: base: components_open: found loaded component usock [nid25257:09727] mca: base: components_open: component usock open function successful [nid25257:09727] mca: base: components_open: found loaded component alps [nid25257:09727] mca: base: components_open: component alps open function successful [nid25257:09727] mca: base: components_open: found loaded component ud [nid25257:09727] mca: base: components_open: component ud open function successful [nid25257:09727] mca: base: components_open: found loaded component tcp [nid25257:09727] mca: base: components_open: component tcp open function successful [nid25257:09727] mca:oob:select: checking available component usock [nid25257:09727] mca:oob:select: Querying component [usock] [nid25257:09727] oob:usock: component_available called [nid25257:09727] [[9128,0],0] USOCK STARTUP [nid25257:09727] SUNPATH: /var/tmp/openmpi-sessions-45504@nid25257_0/9128/0/usock [nid25257:09727] [[9128,0],0] START USOCK LISTENING ON /var/tmp/openmpi-sessions-45504@nid25257_0/9128/0/usock [nid25257:09727] mca:oob:select: Adding component to end [nid25257:09727] mca:oob:select: checking available component alps [nid25257:09727] mca:oob:select: Querying component [alps] [nid25257:09727] mca:oob:select: Skipping component [alps] - no available interfaces [nid25257:09727] mca:oob:select: checking available component ud [nid25257:09727] mca:oob:select: Querying component [ud] [nid25257:09727] oob:ud: component_available called [nid25257:09727] [[9128,0],0] oob:ud:component_init no devices found [nid25257:09727] mca:oob:select: Skipping component [ud] - failed to startup [nid25257:09727] mca:oob:select: checking available component tcp [nid25257:09727] mca:oob:select: Querying component [tcp] [nid25257:09727] oob:tcp: component_available called [nid25257:09727] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [nid25257:09727] [[9128,0],0] oob:tcp:init rejecting loopback interface lo [nid25257:09727] WORKING INTERFACE 2 KERNEL INDEX 1 FAMILY: V4 [nid25257:09727] [[9128,0],0] oob:tcp:init rejecting loopback interface lo [nid25257:09727] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4 [nid25257:09727] [[9128,0],0] oob:tcp:init adding 10.128.99.112 to our list of V4 connections [nid25257:09727] [[9128,0],0] TCP STARTUP [nid25257:09727] [[9128,0],0] attempting to bind to IPv4 port 0 [nid25257:09727] [[9128,0],0] assigned IPv4 port 60755 [nid25257:09727] mca:oob:select: Adding component to end [nid25257:09727] mca:oob:select: Found 2 active transports [nid25257:09727] [[9128,0],0] mca_oob_tcp_listen_thread: new connection: (16, 0) 10.128.69.144:41619 [nid25257:09727] [[9128,0],0] connection_handler: working connection (16, 2) 10.128.69.144:41619 [nid25257:09727] [[9128,0],0] accept_connection: 10.128.69.144:41619 [nid25257:09727] [[9128,0],0]:tcp:recv:handler called [nid25257:09727] [[9128,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 16 [nid25257:09727] [[9128,0],0] waiting for connect ack from UNKNOWN [nid25257:09727] [[9128,0],0] connect ack received from UNKNOWN [nid25257:09727] [[9128,0],0] connect-ack recvd from UNKNOWN [nid25257:09727] [[9128,0],0] mca_oob_tcp_recv_connect: connection from new peer [nid25257:09727] [[9128,0],0] connect-ack header from [[9128,0],2] is okay [nid25257:09727] [[9128,0],0] waiting for connect ack from [[9128,0],2] [nid25257:09727] [[9128,0],0] connect ack received from [[9128,0],2] [nid25257:09727] [[9128,0],0] connect-ack version from [[9128,0],2] matches ours [nid25257:09727] sec: munge validate_cred 12345 [nid25257:09727] sec: munge failed to decode credential: Invalid credential format [nid25257:09727] [[9128,0],0] ORTE_ERROR_LOG: Authentication failed in file ../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803 [nid25257:09727] [[9128,0],0] mca_oob_tcp_listen_thread: new connection: (17, 11) 10.128.69.143:34369 [nid25257:09727] [[9128,0],0] connection_handler: working connection (17, 0) 10.128.69.143:34369 [nid25257:09727] [[9128,0],0] accept_connection: 10.128.69.143:34369 [nid25257:09727] [[9128,0],0]:tcp:recv:handler called [nid25257:09727] [[9128,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 17 [nid25257:09727] [[9128,0],0] waiting for connect ack from UNKNOWN [nid25257:09727] [[9128,0],0] connect ack received from UNKNOWN [nid25257:09727] [[9128,0],0] connect-ack recvd from UNKNOWN [nid25257:09727] [[9128,0],0] mca_oob_tcp_recv_connect: connection from new peer [nid25257:09727] [[9128,0],0] connect-ack header from [[9128,0],1] is okay [nid25257:09727] [[9128,0],0] waiting for connect ack from [[9128,0],1] [nid25257:09727] [[9128,0],0] connect ack received from [[9128,0],1] [nid25257:09727] [[9128,0],0] connect-ack version from [[9128,0],1] matches ours [nid25257:09727] sec: munge validate_cred 12345 [nid25257:09727] sec: munge failed to decode credential: Invalid credential format [nid25257:09727] [[9128,0],0] ORTE_ERROR_LOG: Authentication failed in file ../../../../../orte/mca/oob/tcp/oob_tcp_connection.c at line 803
