You need to increase the wait time, (-w) the default is 120 seconds, you may set it to a higher number like 300-400. The problem is that EC2 takes some time to initiate the machine (which is > 120 seconds sometimes.)
Thanks Best Regards On Wed, Jul 30, 2014 at 8:52 PM, William Cox <[email protected] > wrote: > *TL;DR >50% of the time I can't SSH into either my master or slave nodes > and have to terminate all the machines and restart the EC2 cluster setup > process.* > > Hello, > > I'm trying to setup a Spark cluster on Amazon EC2. I am finding the setup > script to be delicate and unpredictable in terms of reliably allowing SSH > logins to all of the slaves and the master. For instance (I'm running Spark > 0.9.1-hadoop1. since I intend to use Shark. I call this command to > provision a 32 slave cluster using spot instances: > > $./spark-ec2 --spot-price=0.1 --zone=us-east-1e -k key -i ~/key.pem -s 32 >> --instance-type=m1.medium launch cluster_name > > > After waiting for the instances to provision I get the following output: > > All 32 slaves granted >> Launched master in us-east-1e, regid = r-f8444a89 >> Waiting for instances to start up... >> Waiting 120 more seconds... >> Generating cluster's SSH key on master... >> ssh: connect to host ecMASTER.compute-1.amazonaws.com port 22: >> Connection refused >> Error executing remote command, retrying after 30 seconds: Command >> '['ssh', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/user/key.pem', >> '-t', '-t', u'[email protected]', "\n [ -f >> ~/.ssh/id_rsa ] ||\n (ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa >> &&\n cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys)\n "]' >> returned non-zero exit status 255 > > > I have removed the key and machine names with 'MASTER' and 'key'. I wait a > few more cycles of the error message and finally, after 3 attempts, the > script quits with this message: > > > ssh: connect to host ecMASTER.compute-1.amazonaws.com port 22: Connection >> refused >> Error: >> Failed to SSH to remote host ecMASTER.compute-1.amazonaws.com. >> Please check that you have provided the correct --identity-file and >> --key-pair parameters and try again. > > So, YES, the .pem file is correct - I am currently running a smaller > cluster and can provision other machines on EC2 and use that file. > Secondly, the node it can't seem to connect to is the MASTER node. I have > also gone into the EC2 console and verified that all the machines are using > the "key" that corresponds to "key.pem". > > I have tried this command 2x and on a friends machine with no success. > However, I was able to provision a 15 machine cluster using m1.larges. > > Now I PAUSE for some period of time - 2-3 minutes (to write this email) - > and I call the same command with the "--resume" flag. This time it logs > into the master node just fine and begins to give the slaves SSH keys, and > it fails on a certain slave. > >> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22: >> Connection refused >> Error 255 while executing remote command, retrying after 30 seconds >> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22: >> Connection refused >> Error 255 while executing remote command, retrying after 30 seconds >> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22: >> Connection refused >> Error 255 while executing remote command, retrying after 30 seconds >> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22: >> Connection refused >> Traceback (most recent call last): >> File "./spark_ec2.py", line 806, in <module> >> main() >> File "./spark_ec2.py", line 799, in main >> real_main() >> File "./spark_ec2.py", line 684, in real_main >> setup_cluster(conn, master_nodes, slave_nodes, opts, True) >> File "./spark_ec2.py", line 423, in setup_cluster >> ssh_write(slave.public_dns_name, opts, ['tar', 'x'], dot_ssh_tar) >> File "./spark_ec2.py", line 640, in ssh_write >> raise RuntimeError("ssh_write failed with error %s" % proc.returncode) >> RuntimeError: ssh_write failed with error 255 > > So I log into the EC2 console and TERMINATE that specific machine, and > re-resume. Now it finally appears to be installing software on the machines. > > Any ideas why certain machines refuse SSH connections, or why the master > refuses for several minutes and then allows? > > THanks. > > -William > > >
