Re: [OMPI users] This must be ssh problem, but I can't figure out what it is...

Wed, 16 Feb 2011 19:31:23 -0500 

Hi Tena

Again, I think your EC2 session log with ssh debug3 level (below)
should be looked at by somebody more knowledgeable in OpenMPI
and in ssh that me.
There must be some clue to what is going on there.

Ssh experts, Jeff, Ralph, please help!

Anyway ...
AFAIK, 'orted' in the first line you selected/highlighted below,
is the 'Openmpi Run Time Environment Daemon' ( ... the OpenMPI pros
are authorized to send me to the galleys if it is not ...).
So, orted is trying to do its thing, to create the conditions for your
job to run across the two EC2 'instances'. (Gone are the naive
days when these things were computers, each one on its box ...)
This master or ceremonies' work of orted is done via tcp, and I guess 10.96.118.236 is the IP (of computer B?), 
and 56064 is probably the port,
where orted may be trying to open a socket.
The bunch of -mca parameters are just what they are: MCA parameters
(MCA=Modular Component Architecture of OpenMPI, and here I am risking to
be shanghaied or ridiculed again ...).
(You can learn more about the mca parameters with 'ompi_info -help'.)
That is how in my ignorance I parse that line.

So, from the computer/instance-A side orted gives the first kick,
but somehow the ball never comes back from computer/instance-B.
It's ping- without -pong.
The same frustrating feeling I had when I was a kid and kicked the soccer ball on the neighbor's side and would never see it again. 

Cheers,
Gus

Tena Sakai wrote:

Hi Gus,

Thank you for your reply and suggestions.

I will follow up on these in a bit and will give you an
update.  Looking at what vixen and/or dasher generates
from DEBUG3 would be interesting.

For now, may I point out something I noticed out of the
DEBUG3 Output last night?

I found this line:


  debug1: Sending command:  orted --daemonize -mca ess env -mca
orte_ess_jobid 125566976 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 2
--hnp-uri "125566976.0;tcp://10.96.118.236:56064"


Followed by:


  debug2: channel 0: request exec confirm 1
  debug2: fd 3 setting TCP_NODELAY
  debug2: callback done
  debug2: channel 0: open confirm rwindow 0 rmax 32768
  debug3: Wrote 272 bytes for a total of 1893
  debug2: channel 0: rcvd adjust 2097152
  debug2: channel_input_status_confirm: type 99 id 0


It appears, to my untrained eye/mind, a directive from instance A
to B was issued and then what happened?  I don't see that was
honored by the instance B.

Can you please comment on this?

Thank you.

Regards,

Tena

On 2/16/11 1:34 PM, "Gus Correa" <g...@ldeo.columbia.edu> wrote:


Hi Tena

I hope somebody more knowledgeable in ssh
takes a look at the debug3 session log that you included.

I can't see if/where/why ssh is failing for you in EC2.

See other answers inline, please.

Tena Sakai wrote:

Hi Gus,

Thank you again for your reply.


A slight difference is that on vixen and dashen you ran the
MPI hostname tests as a regular user, not as root, right?
Not sure if this will make much of a difference,
but it may be worth trying to run it as a regular user in EC2 also.
I general most people avoid running user applications (MPI programs
included) as root.
Mostly for safety, but I wonder if there are any
implications in the 'rootly powers'
regarding the under-the-hood processes that OpenMPI
launches along with the actual user programs.

Yes, between vixen and dahser I was doing the test as user tsakai,
not as root.  But the reason I wanted to do this test as root is
to show that it fails as regular user (generating pipe system
call failed error), whereas as root it would succeed, as it did
on Friday.

Sorry again.
I even wrote "root can and Tena cannot", then I forgot.
Too many tasks at the same time, too much context-switching ...


The ami has not changed.  The last change on the ami
was last Tuesday.  As such I don't understand this inconsistent
behavior.  I have lots of notes from previous sessions and I
consulted different successful session logs to replicate what I
saw Friday, but with no success.

Having spent days and not getting anywhere, I decided to take a
different approach.  I instantiated a linux ami that was built by
Amazon, which feels like centos/fedora-based.  I downloaded gcc
and c++, plus openMPI 1.4.3.  After I got openMPI running, I
created an account for user tsakai, uploaded my public key, re-logged
in as user tsakai, and ran the same test.  Surprisingly (or not?) it
generated the same result.  I.e., I cannot run the same mpirun
command when there is a remote instance involved, but on itself
mpirun runs fine.  So, I am feeling that this has to be an ssh
authentication problem.  I looked at man page for ssh and ssh_config
and cannot figure out what I am doing wrong.  I put in "LogLevel
DEBUG3" line and it generated lots of lines, in which I found a
line:
  debug1: Authentication succeeded (publickey).
Then I see a bunch of lines that look like:
  debug3: Ignored env XXXXXXX
and mpirun hangs.  Here is the session log:


Ssh on our clusters uses host-based authentication.
I think Reuti sent you his page about it:
http://arc.liv.ac.uk/SGE/howto/hostbased-ssh.html

However, I believe OpenMPI shouldn't care which ssh authentication
mechanism is used, as long as it works passwordless.

As for ssh configuration, ours is pretty standard:

1) We don't have 'IdentitiesOnly yes' (default is 'no'),
but use standard identity file names id_rsa, etc.
I think you are just telling ssh to use the specific identity
file you named.
I don't know if this may cause the problem, but who knows?

2) We don't have 'BatchMode yes' set.

3) We have the GSS authentication set

GSSAPIAuthentication yes

4) The locale environment variables are also passed
(may not be crucial):

        SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY
LC_MESSAGES
        SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
        SendEnv LC_IDENTIFICATION LC_ALL

5) And X forwarding (you're not doing any X stuff, I suppose):

ForwardX11Trusted yes

6) However, you may want to check what is in your
/etc/ssh/ssh_config and /etc/ssh/sshd_config,
because some options may be already set there.

7) Take a look at 'man ssh[d]' and  'man ssh[d]_config' too.

***

Finally, if you are willing to, it may be worth to run the same
experiment (with debug3) on vixen @ dashen, just to compare what
comes out from the verbose ssh messages to what you see in EC2.
Perhaps it may help nail down the reason for failure.

Gus Correa




  [tsakai@vixen ec2]$
  [tsakai@vixen ec2]$ ssh -i $MYKEY
tsa...@ec2-50-17-24-195.compute-1.amazonaws.com
  Last login: Wed Feb 16 06:50:08 2011 from 63.193.205.1

         __|  __|_  )  Amazon Linux AMI
         _|  (     /     Beta
        ___|\___|___|

  See /usr/share/doc/amzn-ami/image-release-notes for latest release notes.
:-)
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # show firewall is off
  [tsakai@domU-12-31-39-16-75-1E ~]$ service iptables status
  -bash: service: command not found
  [tsakai@domU-12-31-39-16-75-1E ~]$ sudo service iptables status
  iptables: Firewall is not running.
  [tsakai@domU-12-31-39-16-75-1E ~]$ # show I can go to inst B with no
password authentication
  [tsakai@domU-12-31-39-16-75-1E ~]$ ssh
domU-12-31-39-16-4E-4C.compute-1.internal
  Last login: Wed Feb 16 06:53:14 2011 from
domu-12-31-39-16-75-1e.compute-1.internal

         __|  __|_  )  Amazon Linux AMI
         _|  (     /     Beta
        ___|\___|___|

  See /usr/share/doc/amzn-ami/image-release-notes for latest release notes.
:-)
  [tsakai@domU-12-31-39-16-4E-4C ~]$
  [tsakai@domU-12-31-39-16-4E-4C ~]$ # also back to inst A
  [tsakai@domU-12-31-39-16-4E-4C ~]$
  [tsakai@domU-12-31-39-16-4E-4C ~]$ ssh
domU-12-31-39-16-75-1E.compute-1.internal
  Last login: Wed Feb 16 06:58:33 2011 from 63.193.205.1

         __|  __|_  )  Amazon Linux AMI
         _|  (     /     Beta
        ___|\___|___|

  See /usr/share/doc/amzn-ami/image-release-notes for latest release notes.
:-)
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # OK
  [tsakai@domU-12-31-39-16-75-1E ~]$ # back to inst B
  [tsakai@domU-12-31-39-16-75-1E ~]$ exit
  logout
  Connection to domU-12-31-39-16-75-1E.compute-1.internal closed.
  [tsakai@domU-12-31-39-16-4E-4C ~]$
  [tsakai@domU-12-31-39-16-4E-4C ~]$ env | grep LD_LIB
  LD_LIBRARY_PATH=:/usr/local/lib
  [tsakai@domU-12-31-39-16-4E-4C ~]$ # show no firewall on inst B
  [tsakai@domU-12-31-39-16-4E-4C ~]$ sudo service iptables status
  iptables: Firewall is not running.
  [tsakai@domU-12-31-39-16-4E-4C ~]$
  [tsakai@domU-12-31-39-16-4E-4C ~]$ # go back to inst A
  [tsakai@domU-12-31-39-16-4E-4C ~]$ exit
  logout
  Connection to domU-12-31-39-16-4E-4C.compute-1.internal closed.
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ env | grep LD_LIB
  LD_LIBRARY_PATH=:/usr/local/lib
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ cat app.ac
  -H domU-12-31-39-16-75-1E.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-16-75-1E.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-16-4E-4C.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-16-4E-4C.compute-1.internal -np 1 /bin/hostname
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # top 2 are inst A (this machine);
bottom 2 are remote inst (inst B)
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ mpirun -app app.ac
  ^Cmpirun: killing job...

  --------------------------------------------------------------------------
  mpirun noticed that the job aborted, but has no info as to the process
  that caused that situation.
  --------------------------------------------------------------------------
  --------------------------------------------------------------------------
  mpirun was unable to cleanly terminate the daemons on the nodes shown
  below. Additional manual cleanup may be required - please refer to
  the "orte-clean" tool for assistance.
  --------------------------------------------------------------------------
        domU-12-31-39-16-4E-4C.compute-1.internal - daemon did not report
back when launched
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # *** daemon did not report back when
launched ***
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ cat app.ac2
  -H domU-12-31-39-16-75-1E.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-16-75-1E.compute-1.internal -np 1 /bin/hostname
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # they refer to this instance (inst A)
  [tsakai@domU-12-31-39-16-75-1E ~]$ mpirun -app app.ac2
  domU-12-31-39-16-75-1E
  domU-12-31-39-16-75-1E
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # that's no problem
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ cd .ssh
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ cat config
  Host *
        IdentityFile /home/tsakai/.ssh/tsakai
        IdentitiesOnly yes
        BatchMode yes
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ mv config config.svd
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ cat config.svd > config
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ ll config
  -rw-rw-r-- 1 tsakai tsakai 81 Feb 16 07:06 config
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ chmod 600 config
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ cat config
  Host *
        IdentityFile /home/tsakai/.ssh/tsakai
        IdentitiesOnly yes
        BatchMode yes
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ cat - >> config
        LogLevel DEBUG3
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ cat config
  Host *
        IdentityFile /home/tsakai/.ssh/tsakai
        IdentitiesOnly yes
        BatchMode yes
        LogLevel DEBUG3
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ ll config
  -rw------- 1 tsakai tsakai 98 Feb 16 07:07 config
  [tsakai@domU-12-31-39-16-75-1E .ssh]$
  [tsakai@domU-12-31-39-16-75-1E .ssh]$ cd ..
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ mpirun -app app.ac
  debug2: ssh_connect: needpriv 0
  debug1: Connecting to domU-12-31-39-16-4E-4C.compute-1.internal
[10.96.77.182] port 22.
  debug1: Connection established.
  debug3: Not a RSA1 key file /home/tsakai/.ssh/tsakai.
  debug2: key_type_from_name: unknown key type '-----BEGIN'
  debug3: key_read: missing keytype
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug3: key_read: missing whitespace
  debug2: key_type_from_name: unknown key type '-----END'
  debug3: key_read: missing keytype
  debug1: identity file /home/tsakai/.ssh/tsakai type -1
  debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
  debug1: match: OpenSSH_5.3 pat OpenSSH*
  debug1: Enabling compatibility mode for protocol 2.0
  debug1: Local version string SSH-2.0-OpenSSH_5.3
  debug2: fd 3 setting O_NONBLOCK
  debug1: SSH2_MSG_KEXINIT sent
  debug3: Wrote 792 bytes for a total of 813
  debug1: SSH2_MSG_KEXINIT received
  debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diff
ie-hellman-group14-sha1,diffie-hellman-group1-sha1
  debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
  debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,b
lowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.l
iu.se
  debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,b
lowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.l
iu.se
  debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac...@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh
.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac...@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh
.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: none,z...@openssh.com,zlib
  debug2: kex_parse_kexinit: none,z...@openssh.com,zlib
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit: first_kex_follows 0
  debug2: kex_parse_kexinit: reserved 0
  debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diff
ie-hellman-group14-sha1,diffie-hellman-group1-sha1
  debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
  debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,b
lowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.l
iu.se
  debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,b
lowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.l
iu.se
  debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac...@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh
.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac...@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh
.com,hmac-sha1-96,hmac-md5-96
  debug2: kex_parse_kexinit: none,z...@openssh.com
  debug2: kex_parse_kexinit: none,z...@openssh.com
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit:
  debug2: kex_parse_kexinit: first_kex_follows 0
  debug2: kex_parse_kexinit: reserved 0
  debug2: mac_setup: found hmac-md5
  debug1: kex: server->client aes128-ctr hmac-md5 none
  debug2: mac_setup: found hmac-md5
  debug1: kex: client->server aes128-ctr hmac-md5 none
  debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
  debug3: Wrote 24 bytes for a total of 837
  debug2: dh_gen_key: priv key bits set: 125/256
  debug2: bits set: 489/1024
  debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
  debug3: Wrote 144 bytes for a total of 981
  debug3: check_host_in_hostfile: filename /home/tsakai/.ssh/known_hosts
  debug3: check_host_in_hostfile: match line 1
  debug3: check_host_in_hostfile: filename /home/tsakai/.ssh/known_hosts
  debug3: check_host_in_hostfile: match line 1
  debug1: Host 'domu-12-31-39-16-4e-4c.compute-1.internal' is known and
matches the RSA host key.
  debug1: Found key in /home/tsakai/.ssh/known_hosts:1
  debug2: bits set: 491/1024
  debug1: ssh_rsa_verify: signature correct
  debug2: kex_derive_keys
  debug2: set_newkeys: mode 1
  debug1: SSH2_MSG_NEWKEYS sent
  debug1: expecting SSH2_MSG_NEWKEYS
  debug3: Wrote 16 bytes for a total of 997
  debug2: set_newkeys: mode 0
  debug1: SSH2_MSG_NEWKEYS received
  debug1: SSH2_MSG_SERVICE_REQUEST sent
  debug3: Wrote 48 bytes for a total of 1045
  debug2: service_accept: ssh-userauth
  debug1: SSH2_MSG_SERVICE_ACCEPT received
  debug2: key: /home/tsakai/.ssh/tsakai ((nil))
  debug3: Wrote 64 bytes for a total of 1109
  debug1: Authentications that can continue: publickey
  debug3: start over, passed a different list publickey
  debug3: preferred gssapi-with-mic,publickey
  debug3: authmethod_lookup publickey
  debug3: remaining preferred: ,publickey
  debug3: authmethod_is_enabled publickey
  debug1: Next authentication method: publickey
  debug1: Trying private key: /home/tsakai/.ssh/tsakai
  debug1: read PEM private key done: type RSA
  debug3: sign_and_send_pubkey
  debug2: we sent a publickey packet, wait for reply
  debug3: Wrote 384 bytes for a total of 1493
  debug1: Authentication succeeded (publickey).
  debug2: fd 4 setting O_NONBLOCK
  debug1: channel 0: new [client-session]
  debug3: ssh_session2_open: channel_new: 0
  debug2: channel 0: send open
  debug1: Requesting no-more-sessi...@openssh.com
  debug1: Entering interactive session.
  debug3: Wrote 128 bytes for a total of 1621
  debug2: callback start
  debug2: client_session2_setup: id 0
  debug1: Sending environment.
  debug3: Ignored env HOSTNAME
  debug3: Ignored env TERM
  debug3: Ignored env SHELL
  debug3: Ignored env HISTSIZE
  debug3: Ignored env EC2_AMITOOL_HOME
  debug3: Ignored env SSH_CLIENT
  debug3: Ignored env SSH_TTY
  debug3: Ignored env USER
  debug3: Ignored env LD_LIBRARY_PATH
  debug3: Ignored env LS_COLORS
  debug3: Ignored env EC2_HOME
  debug3: Ignored env MAIL
  debug3: Ignored env PATH
  debug3: Ignored env INPUTRC
  debug3: Ignored env PWD
  debug3: Ignored env JAVA_HOME
  debug1: Sending env LANG = en_US.UTF-8
  debug2: channel 0: request env confirm 0
  debug3: Ignored env AWS_CLOUDWATCH_HOME
  debug3: Ignored env AWS_IAM_HOME
  debug3: Ignored env SHLVL
  debug3: Ignored env HOME
  debug3: Ignored env AWS_PATH
  debug3: Ignored env AWS_AUTO_SCALING_HOME
  debug3: Ignored env LOGNAME
  debug3: Ignored env AWS_ELB_HOME
  debug3: Ignored env SSH_CONNECTION
  debug3: Ignored env LESSOPEN
  debug3: Ignored env AWS_RDS_HOME
  debug3: Ignored env G_BROKEN_FILENAMES
  debug3: Ignored env _
  debug3: Ignored env OLDPWD
  debug3: Ignored env OMPI_MCA_plm
  debug1: Sending command:  orted --daemonize -mca ess env -mca
orte_ess_jobid 125566976 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 2
--hnp-uri "125566976.0;tcp://10.96.118.236:56064"
  debug2: channel 0: request exec confirm 1
  debug2: fd 3 setting TCP_NODELAY
  debug2: callback done
  debug2: channel 0: open confirm rwindow 0 rmax 32768
  debug3: Wrote 272 bytes for a total of 1893
  debug2: channel 0: rcvd adjust 2097152
  debug2: channel_input_status_confirm: type 99 id 0
  debug2: exec request accepted on channel 0
  debug2: channel 0: read<=0 rfd 4 len 0
  debug2: channel 0: read failed
  debug2: channel 0: close_read
  debug2: channel 0: input open -> drain
  debug2: channel 0: ibuf empty
  debug2: channel 0: send eof
  debug2: channel 0: input drain -> closed
  debug3: Wrote 32 bytes for a total of 1925
  debug2: channel 0: rcvd eof
  debug2: channel 0: output open -> drain
  debug2: channel 0: obuf empty
  debug2: channel 0: close_write
  debug2: channel 0: output drain -> closed
  debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
  debug2: channel 0: rcvd close
  debug3: channel 0: will not send data after close
  debug2: channel 0: almost dead
  debug2: channel 0: gc: notify user
  debug2: channel 0: gc: user detached
  debug2: channel 0: send close
  debug2: channel 0: is dead
  debug2: channel 0: garbage collecting
  debug1: channel 0: free: client-session, nchannels 1
  debug3: channel 0: status: The following connections are open:
    #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)

  debug3: channel 0: close_fds r -1 w -1 e 6 c -1
  debug3: Wrote 32 bytes for a total of 1957
  debug3: Wrote 64 bytes for a total of 2021
  debug1: fd 0 clearing O_NONBLOCK
  Transferred: sent 1840, received 1896 bytes, in 0.1 seconds
  Bytes per second: sent 18384.8, received 18944.3
  debug1: Exit status 0
  # it is hanging; I am about to issue control-C
  ^Cmpirun: killing job...

  --------------------------------------------------------------------------
  mpirun noticed that the job aborted, but has no info as to the process
  that caused that situation.
  --------------------------------------------------------------------------
  --------------------------------------------------------------------------
  mpirun was unable to cleanly terminate the daemons on the nodes shown
  below. Additional manual cleanup may be required - please refer to
  the "orte-clean" tool for assistance.
  --------------------------------------------------------------------------
        domU-12-31-39-16-4E-4C.compute-1.internal - daemon did not report
back when launched
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # it says the same thing, i.e.,
  [tsakai@domU-12-31-39-16-75-1E ~]$ # daemon did not report back when
launched
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # what does that mean?
  [tsakai@domU-12-31-39-16-75-1E ~]$ # ssh doesn't say anything alarming...
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ # I give up
  [tsakai@domU-12-31-39-16-75-1E ~]$
  [tsakai@domU-12-31-39-16-75-1E ~]$ exit
  logout
  [tsakai@vixen ec2]$
  [tsakai@vixen ec2]$

Do you see anything strange?

One final question: On ssh man page, it mentions a few environmental
varialbles.  SSH_ASKPASS, SSH_AUTH_SOCK, SSH_CONNECTION, etc.  Do
any of these matter as far as openMPI is concerned?

Thank you, Gus.

Regards,

Tena

On 2/15/11 5:09 PM, "Gus Correa" <g...@ldeo.columbia.edu> wrote:


Tena Sakai wrote:

Hi,

I am trying to reproduce what I was able to show last Friday on Amazon
EC2 instances, but I am having a problem.  What I was able to show last
Friday as root was with this command:
  mpirun ­app app.ac
with app.ac being:
  -H dns-entry-A ­np 1 (linux command)
  -H dns-entry-A ­np 1 (linux command)
  -H dns-entry-B ­np 1 (linux command)
  -H dns-entry-B ­np 1 (linux command)

Here¹s the config file in root¹s .ssh directory:
  Host *
        IdentityFile /root/.ssh/.derobee/.kagi
        IdentitiesOnly yes
        BatchMode yes

Yesterday and today I can¹t get this to work.  I made the last part of
app.ac
file simpler (it now says /bin/hostname).  Below is the session:

  -bash-3.2#
  -bash-3.2# # I am on instance A, host name for inst A is:
  -bash-3.2# hostname
  domU-12-31-39-09-CD-C2
  -bash-3.2#
  -bash-3.2# nslookup domU-12-31-39-09-CD-C2
  Server:               172.16.0.23
  Address:      172.16.0.23#53

  Non-authoritative answer:
  Name: domU-12-31-39-09-CD-C2.compute-1.internal
  Address: 10.210.210.48

  -bash-3.2# cd .ssh
  -bash-3.2#
  -bash-3.2# cat config
  Host *
          IdentityFile /root/.ssh/.derobee/.kagi
          IdentitiesOnly yes
          BatchMode yes
  -bash-3.2#
  -bash-3.2# ll config
  -rw-r--r-- 1 root root 103 Feb 15 17:18 config
  -bash-3.2#
  -bash-3.2# chmod 600 config
  -bash-3.2#
  -bash-3.2# # show I can go to inst B without password/passphrase
  -bash-3.2#
  -bash-3.2# ssh domU-12-31-39-09-E6-71.compute-1.internal
  Last login: Tue Feb 15 17:18:46 2011 from 10.210.210.48
  -bash-3.2#
  -bash-3.2# hostname
  domU-12-31-39-09-E6-71
  -bash-3.2#
  -bash-3.2# nslookup `hostname`
  Server:               172.16.0.23
  Address:      172.16.0.23#53

  Non-authoritative answer:
  Name: domU-12-31-39-09-E6-71.compute-1.internal
  Address: 10.210.233.123

  -bash-3.2# # and back to inst A is also no problem
  -bash-3.2#
  -bash-3.2# ssh domU-12-31-39-09-CD-C2.compute-1.internal
  Last login: Tue Feb 15 17:36:19 2011 from 63.193.205.1
  -bash-3.2#
  -bash-3.2# hostname
  domU-12-31-39-09-CD-C2
  -bash-3.2#
  -bash-3.2# # log out twice to go back to inst A
  -bash-3.2# exit
  logout
  Connection to domU-12-31-39-09-CD-C2.compute-1.internal closed.
  -bash-3.2#
  -bash-3.2# exit
  logout
  Connection to domU-12-31-39-09-E6-71.compute-1.internal closed.
  -bash-3.2#
  -bash-3.2# hostname
  domU-12-31-39-09-CD-C2
  -bash-3.2#
  -bash-3.2# cd ..
  -bash-3.2#
  -bash-3.2# pwd
  /root
  -bash-3.2#
  -bash-3.2# ll
  total 8
  -rw-r--r-- 1 root root 260 Feb 15 17:24 app.ac
  -rw-r--r-- 1 root root 130 Feb 15 17:34 app.ac2
  -bash-3.2#
  -bash-3.2# cat app.ac
  -H domU-12-31-39-09-CD-C2.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-09-CD-C2.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-09-E6-71.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-09-E6-71.compute-1.internal -np 1 /bin/hostname
  -bash-3.2#
  -bash-3.2# # when there is a remote machine (bottome 2 lines) it hangs
  -bash-3.2# mpirun -app app.ac
  mpirun: killing job...


--------------------------------------------------------------------------
  mpirun noticed that the job aborted, but has no info as to the process
  that caused that situation.

--------------------------------------------------------------------------

--------------------------------------------------------------------------
  mpirun was unable to cleanly terminate the daemons on the nodes shown
  below. Additional manual cleanup may be required - please refer to
  the "orte-clean" tool for assistance.

--------------------------------------------------------------------------
        domU-12-31-39-09-E6-71.compute-1.internal - daemon did not
report back when launched
  -bash-3.2#
  -bash-3.2# cat app.ac2
  -H domU-12-31-39-09-CD-C2.compute-1.internal -np 1 /bin/hostname
  -H domU-12-31-39-09-CD-C2.compute-1.internal -np 1 /bin/hostname
  -bash-3.2#
  -bash-3.2# # when there is no remote machine, then mpirun works:
  -bash-3.2# mpirun -app app.ac2
  domU-12-31-39-09-CD-C2
  domU-12-31-39-09-CD-C2
  -bash-3.2#
  -bash-3.2# hostname
  domU-12-31-39-09-CD-C2
  -bash-3.2#
  -bash-3.2# # this gotta be ssh problem....
  -bash-3.2#
  -bash-3.2# # show no firewall is used
  -bash-3.2# iptables --list
  Chain INPUT (policy ACCEPT)
   target     prot opt source               destination

  Chain FORWARD (policy ACCEPT)
  target     prot opt source               destination

  Chain OUTPUT (policy ACCEPT)
  target     prot opt source               destination
  -bash-3.2#
  -bash-3.2# exit
  logout
  [tsakai@vixen ec2]$

Would someone please point out what I am doing wrong?

Thank you.

Regards,

Tena


Hi Tena

Nothing wrong that I can see.
Just another couple of suggestions,
based on somewhat vague possibilities.

A slight difference is that on vixen and dashen you ran the
MPI hostname tests as a regular user, not as root, right?
Not sure if this will make much of a difference,
but it may be worth trying to run it as a regular user in EC2 also.
I general most people avoid running user applications (MPI programs
included) as root.
Mostly for safety, but I wonder if there are any
implications in the 'rootly powers'
regarding the under-the-hood processes that OpenMPI
launches along with the actual user programs.

This may make no difference either,
but you could do a 'service iptables status',
to see if the service is running, even though there are
no explicit iptable rules (as per your email).
If the service is not running you get
'Firewall is stopped.' (in CentOS).
I *think* 'iptables --list' loads the iptables module into the
kernel, as a side effect, whereas the service command does not.
So, it may be cleaner (safer?) to use the service version
instead of 'iptables --list'.
I don't know if it will make any difference,
but just in case, if the service is running,
why not do 'service iptables stop',
and perhaps also 'chkconfig iptables off' to be completely
free of iptables?

Gus Correa
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Reply via email to