Script started on Thu 10 Feb 2011 11:39:04 AM PST [tsakai@vixen ec2]$ [tsakai@vixen ec2]$ tty /dev/pts/7 [tsakai@vixen ec2]$ pwd /home/tsakai/Notes/R/parallel/Rmpi/ec2 [tsakai@vixen ec2]$ [tsakai@vixen ec2]$ 2ec2 ec2-174-129-153-175.compute-1.amazonaws.com The authenticity of host 'ec2-174-129-153-175.compute-1.amazonaws.com (174.129.153.175)' can't be established. RSA key fingerprint is e3:ad:75:b1:a4:63:7f:0f:c4:0b:10:71:f3:2f:21:81. Are you sure you want to continue connecting (yes/no)? yes Last login: Tue Feb 8 22:52:54 2011 from 10.201.197.188 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # I am on machine A [tsakai@ip-10-194-95-239 ~]$ # whose hostname is: [tsakai@ip-10-194-95-239 ~]$ hostname ip-10-194-95-239 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # I know machine B's private dns as 'ip-10-114-133-231.ec2.internal' [tsakai@ip-10-194-95-239 ~]$ # from amazon's management console. About to connect from A to B: [tsakai@ip-10-194-95-239 ~]$ ssh ip-10-114-133-231 The authenticity of host 'ip-10-114-133-231 (10.114.133.231)' can't be established. RSA key fingerprint is e3:ad:75:b1:a4:63:7f:0f:c4:0b:10:71:f3:2f:21:81. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ip-10-114-133-231,10.114.133.231' (RSA) to the list of known hosts. tsakai@ip-10-114-133-231's password: Permission denied, please try again. tsakai@ip-10-114-133-231's password: [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # that failed, I think, because I didn't use -i flag. [tsakai@ip-10-194-95-239 ~]$ # this time with -i flag: [tsakai@ip-10-194-95-239 ~]$ ssh -i .ssh/tsakai ip-10-114-133-231 Last login: Tue Feb 8 22:52:54 2011 from 10.201.197.188 [tsakai@ip-10-114-133-231 ~]$ [tsakai@ip-10-114-133-231 ~]$ # yep, I am on machine B. [tsakai@ip-10-114-133-231 ~]$ # let me double check: [tsakai@ip-10-114-133-231 ~]$ hostname ip-10-114-133-231 [tsakai@ip-10-114-133-231 ~]$ [tsakai@ip-10-114-133-231 ~]$ # now, how about B to A? [tsakai@ip-10-114-133-231 ~]$ # first without -i flag [tsakai@ip-10-114-133-231 ~]$ ssh ip-10-194-95-239 The authenticity of host 'ip-10-194-95-239 (10.194.95.239)' can't be established. RSA key fingerprint is e3:ad:75:b1:a4:63:7f:0f:c4:0b:10:71:f3:2f:21:81. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ip-10-194-95-239,10.194.95.239' (RSA) to the list of known hosts. tsakai@ip-10-194-95-239's password: Permission denied, please try again. tsakai@ip-10-194-95-239's password: [tsakai@ip-10-114-133-231 ~]$ # no, I need -i flag. [tsakai@ip-10-114-133-231 ~]$ ssh -i .ssh/tsakai ip-10-194-95-239 Last login: Thu Feb 10 14:39:41 2011 from 63.193.205.1 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # that too works. [tsakai@ip-10-194-95-239 ~]$ exit logout Connection to ip-10-194-95-239 closed. [tsakai@ip-10-114-133-231 ~]$ # I am on machine B [tsakai@ip-10-114-133-231 ~]$ # I want to change .ssh/config file as reuti suggested [tsakai@ip-10-114-133-231 ~]$ cd .ssh [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ cat config cat: config: No such file or directory [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ ll -a total 20 drwx------ 2 tsakai tsakai 4096 Feb 8 22:13 . drwx------ 3 tsakai tsakai 4096 Feb 5 23:23 .. -rw------- 1 tsakai tsakai 232 Feb 5 23:19 authorized_keys -rw-r--r-- 1 tsakai tsakai 1287 Feb 10 14:46 known_hosts -rw------- 1 tsakai tsakai 887 Feb 8 22:03 tsakai [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ # I see, I just launched this instance and therefore there is no config here. [tsakai@ip-10-114-133-231 .ssh]$ cat - > config Host * IdentityFile /home/tsakai/.ssh/tsakai IdentitiesOnly yes BatchMode yes LogLevel DEBUG3 [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ cat config Host * IdentityFile /home/tsakai/.ssh/tsakai IdentitiesOnly yes BatchMode yes LogLevel DEBUG3 [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ # that looks OK, scp it to machine A [tsakai@ip-10-114-133-231 .ssh]$ scp -i ./tsakai config ip-10-194-95-239.ec2:/home/tsakai/.ssh Bad owner or permissions on /home/tsakai/.ssh/config lost connection [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ # OK... [tsakai@ip-10-114-133-231 .ssh]$ ll config -rw-rw-r-- 1 tsakai tsakai 98 Feb 10 14:51 config [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ chmod 600 config [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ scp -i ./tsakai config ip-10-194-95-239.ec2:/home/tsakai/.ssh debug2: ssh_connect: needpriv 0 ssh: ip-10-194-95-239.ec2: Name or service not known lost connection [tsakai@ip-10-114-133-231 .ssh]$ # that .ec2 is the problem. [tsakai@ip-10-114-133-231 .ssh]$ scp -i ./tsakai config ip-10-194-95-239:/home/tsakai/.ssh debug2: ssh_connect: needpriv 0 debug1: Connecting to ip-10-194-95-239 [10.194.95.239] port 22. debug1: Connection established. debug3: Not a RSA1 key file ./tsakai. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file ./tsakai type -1 debug3: Not a RSA1 key file /home/tsakai/.ssh/tsakai. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /home/tsakai/.ssh/tsakai type -1 debug1: loaded 2 keys debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3 debug1: match: OpenSSH_4.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.3 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_init: found hmac-md5 debug1: kex: server->client aes128-cbc hmac-md5 none debug2: mac_init: found hmac-md5 debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug2: dh_gen_key: priv key bits set: 127/256 debug2: bits set: 519/1024 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: check_host_in_hostfile: filename /home/tsakai/.ssh/known_hosts debug3: check_host_in_hostfile: match line 3 debug3: check_host_in_hostfile: filename /home/tsakai/.ssh/known_hosts debug3: check_host_in_hostfile: match line 3 debug1: Host 'ip-10-194-95-239' is known and matches the RSA host key. debug1: Found key in /home/tsakai/.ssh/known_hosts:3 debug2: bits set: 528/1024 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: ./tsakai ((nil)) debug2: key: /home/tsakai/.ssh/tsakai ((nil)) debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-with-mic,password debug3: preferred gssapi-with-mic,publickey debug3: authmethod_lookup gssapi-with-mic debug3: remaining preferred: publickey debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug3: Trying to reverse map address 10.194.95.239. debug1: Unspecified GSS failure. Minor code may provide more information Unknown code krb5 195 debug1: Unspecified GSS failure. Minor code may provide more information Unknown code krb5 195 debug1: Unspecified GSS failure. Minor code may provide more information Unknown code krb5 195 debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: ./tsakai debug1: read PEM private key done: type RSA debug3: sign_and_send_pubkey debug2: we sent a publickey packet, wait for reply debug1: Authentication succeeded (publickey). debug2: fd 4 setting O_NONBLOCK debug2: fd 5 setting O_NONBLOCK debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug1: Entering interactive session. debug2: callback start debug2: client_session2_setup: id 0 debug1: Sending environment. debug3: Ignored env HOSTNAME debug3: Ignored env TERM debug3: Ignored env SHELL debug3: Ignored env HISTSIZE debug3: Ignored env SSH_CLIENT debug3: Ignored env SSH_TTY debug3: Ignored env USER debug3: Ignored env LS_COLORS debug3: Ignored env MAIL debug3: Ignored env PATH debug3: Ignored env INPUTRC debug3: Ignored env PWD debug3: Ignored env SSH_ASKPASS debug3: Ignored env SHLVL debug3: Ignored env HOME debug3: Ignored env LOGNAME debug3: Ignored env CVS_RSH debug3: Ignored env SSH_CONNECTION debug3: Ignored env LESSOPEN debug3: Ignored env G_BROKEN_FILENAMES debug3: Ignored env _ debug3: Ignored env OLDPWD debug1: Sending command: scp -t /home/tsakai/.ssh debug2: channel 0: request exec confirm 0 debug2: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 config 0% 0 0.0KB/s --:-- ETAconfig 100% 98 0.1KB/s 00:00 debug2: channel 0: read<=0 rfd 4 len 0 debug2: channel 0: read failed debug2: channel 0: close_read debug2: channel 0: input open -> drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug2: channel 0: input drain -> closed debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1) debug3: channel 0: close_fds r -1 w -1 e 6 c -1 debug1: fd 0 clearing O_NONBLOCK debug1: fd 1 clearing O_NONBLOCK debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0 debug1: Exit status 0 [tsakai@ip-10-114-133-231 .ssh]$ [tsakai@ip-10-114-133-231 .ssh]$ # Wow, that's a lot of info, but I think scp went fine. [tsakai@ip-10-114-133-231 .ssh]$ # go back to machine A. [tsakai@ip-10-114-133-231 .ssh]$ exit logout Connection to ip-10-114-133-231 closed. [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ hostname ip-10-194-95-239 [tsakai@ip-10-194-95-239 ~]$ # I am indeed on host machine A. [tsakai@ip-10-194-95-239 ~]$ cd .ssh [tsakai@ip-10-194-95-239 .ssh]$ [tsakai@ip-10-194-95-239 .ssh]$ ll total 16 -rw------- 1 tsakai tsakai 232 Feb 5 23:19 authorized_keys -rw------- 1 tsakai tsakai 98 Feb 10 14:55 config -rw-r--r-- 1 tsakai tsakai 1289 Feb 10 14:43 known_hosts -rw------- 1 tsakai tsakai 887 Feb 8 22:03 tsakai [tsakai@ip-10-194-95-239 .ssh]$ [tsakai@ip-10-194-95-239 .ssh]$ cat config Host * IdentityFile /home/tsakai/.ssh/tsakai IdentitiesOnly yes BatchMode yes LogLevel DEBUG3 [tsakai@ip-10-194-95-239 .ssh]$ # config file arrived and looks good. [tsakai@ip-10-194-95-239 .ssh]$ [tsakai@ip-10-194-95-239 .ssh]$ cd .. [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ ll total 0 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # I just uploaded 2 files from my local machine, vixen. [tsakai@ip-10-194-95-239 ~]$ ll total 8 -rw-r--r-- 1 tsakai tsakai 320 Feb 10 15:01 app.ac2 -rwxr-xr-x 1 tsakai tsakai 379 Feb 10 15:01 fib.R [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ cat fib.R # fib() computes, given index n, fibonacci number iteratively # here's the first dozen sequence (indexed from 0..11) # 1, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 fib <- function( n ) { a <- 0 b <- 1 for ( i in 1:n ) { t <- b b <- a a <- a + t } a } arg <- commandArgs( TRUE ) myHost <- system( 'hostname', intern=TRUE ) cat( fib(arg), myHost, '\n' ) [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ cat app.ac2 -H domU-12-31-39-06-74-E2.compute-1.internal -np 1 Rscript /home/tsakai/fib.R 5 -H domU-12-31-39-06-74-E2.compute-1.internal -np 1 Rscript /home/tsakai/fib.R 6 -H domU-12-31-39-07-35-21.compute-1.internal -np 1 Rscript /home/tsakai/fib.R 7 -H domU-12-31-39-07-35-21.compute-1.internal -np 1 Rscript /home/tsakai/fib.R 8 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # oops, they are wrong addresses [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ cat - > app.ac2 -H ip-10-194-95-239 -np 1 Rscript /home/tsakai/fib.R 5 -H ip-10-194-95-239 -np 1 Rscript /home/tsakai/fib.R 6 -H ip-10-114-133-231 -np 1 Rscript /home/tsakai/fib.R 7 -H ip-10-114-133-231 -np 1 Rscript /home/tsakai/fib.R 8 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ cat app.ac2 -H ip-10-194-95-239 -np 1 Rscript /home/tsakai/fib.R 5 -H ip-10-194-95-239 -np 1 Rscript /home/tsakai/fib.R 6 -H ip-10-114-133-231 -np 1 Rscript /home/tsakai/fib.R 7 -H ip-10-114-133-231 -np 1 Rscript /home/tsakai/fib.R 8 [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # that looks oK. Run mpirun command and see what happens... [tsakai@ip-10-194-95-239 ~]$ mpirun -app app.ac2 debug2: ssh_connect: needpriv 0 debug1: Connecting to ip-10-114-133-231 [10.114.133.231] port 22. debug1: Connection established. debug3: Not a RSA1 key file /home/tsakai/.ssh/tsakai. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /home/tsakai/.ssh/tsakai type -1 debug1: loaded 1 keys debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3 debug1: match: OpenSSH_4.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.3 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_init: found hmac-md5 debug1: kex: server->client aes128-cbc hmac-md5 none debug2: mac_init: found hmac-md5 debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug2: dh_gen_key: priv key bits set: 128/256 debug2: bits set: 529/1024 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: check_host_in_hostfile: filename /home/tsakai/.ssh/known_hosts debug3: check_host_in_hostfile: match line 3 debug3: check_host_in_hostfile: filename /home/tsakai/.ssh/known_hosts debug3: check_host_in_hostfile: match line 3 debug1: Host 'ip-10-114-133-231' is known and matches the RSA host key. debug1: Found key in /home/tsakai/.ssh/known_hosts:3 debug2: bits set: 508/1024 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /home/tsakai/.ssh/tsakai ((nil)) debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-with-mic,password debug3: preferred gssapi-with-mic,publickey debug3: authmethod_lookup gssapi-with-mic debug3: remaining preferred: publickey debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug3: Trying to reverse map address 10.114.133.231. debug1: Unspecified GSS failure. Minor code may provide more information Unknown code krb5 195 debug1: Unspecified GSS failure. Minor code may provide more information Unknown code krb5 195 debug1: Unspecified GSS failure. Minor code may provide more information Unknown code krb5 195 debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/tsakai/.ssh/tsakai debug1: read PEM private key done: type RSA debug3: sign_and_send_pubkey debug2: we sent a publickey packet, wait for reply debug1: Authentication succeeded (publickey). debug2: fd 4 setting O_NONBLOCK debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug1: Entering interactive session. debug2: callback start debug2: client_session2_setup: id 0 debug1: Sending environment. debug3: Ignored env HOSTNAME debug3: Ignored env TERM debug3: Ignored env SHELL debug3: Ignored env HISTSIZE debug3: Ignored env SSH_CLIENT debug3: Ignored env SSH_TTY debug3: Ignored env USER debug3: Ignored env LS_COLORS debug3: Ignored env MAIL debug3: Ignored env PATH debug3: Ignored env INPUTRC debug3: Ignored env PWD debug3: Ignored env SSH_ASKPASS debug3: Ignored env SHLVL debug3: Ignored env HOME debug3: Ignored env LOGNAME debug3: Ignored env CVS_RSH debug3: Ignored env SSH_CONNECTION debug3: Ignored env LESSOPEN debug3: Ignored env G_BROKEN_FILENAMES debug3: Ignored env _ debug3: Ignored env OLDPWD debug3: Ignored env OMPI_MCA_plm debug1: Sending command: orted --daemonize -mca ess env -mca orte_ess_jobid 3344891904 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 2 --hnp-uri "3344891904.0;tcp://10.194.95.239:54256" debug2: channel 0: request exec confirm 0 debug2: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug2: channel 0: read<=0 rfd 4 len 0 debug2: channel 0: read failed debug2: channel 0: close_read debug2: channel 0: input open -> drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug2: channel 0: input drain -> closed debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1) debug3: channel 0: close_fds r -1 w -1 e 6 c -1 debug1: fd 0 clearing O_NONBLOCK debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.2 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0 debug1: Exit status 0 mpirun: killing job... -------------------------------------------------------------------------- mpirun noticed that the job aborted, but has no info as to the process that caused that situation. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun was unable to cleanly terminate the daemons on the nodes shown below. Additional manual cleanup may be required - please refer to the "orte-clean" tool for assistance. -------------------------------------------------------------------------- ip-10-114-133-231 - daemon did not report back when launched [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # It hung and after approx 30 seconds I controled-C'ed. [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ # Let me get off this instance and [tsakai@ip-10-194-95-239 ~]$ # get a printout of the debugging messages and read [tsakai@ip-10-194-95-239 ~]$ # and send the whole session to the list. [tsakai@ip-10-194-95-239 ~]$ [tsakai@ip-10-194-95-239 ~]$ exit logout [tsakai@vixen ec2]$ [tsakai@vixen ec2]$ exit exit Script done on Thu 10 Feb 2011 12:10:40 PM PST