To add to this: for this many (>= 20) machines I usually use at least
--wait 600.
On Wed, Jul 30, 2014 at 9:10 AM, Nicholas Chammas
<nicholas.cham...@gmail.com> wrote:
> William,
>
> The error you are seeing is misleading. There is no need to terminate the
> cluster and start over.
>
> Just re-run your launch command, but with the additional --resume option
> tacked on the end.
>
> As Akhil explained, this happens because AWS is not starting up the
> instances as quickly as the script is expecting. You can increase the wait
> time to mitigate this problem.
>
> Nick
>
>
>
> On Wed, Jul 30, 2014 at 11:51 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>>
>> You need to increase the wait time, (-w) the default is 120 seconds, you
>> may set it to a higher number like 300-400. The problem is that EC2 takes
>> some time to initiate the machine (which is > 120 seconds sometimes.)
>>
>> Thanks
>> Best Regards
>>
>>
>> On Wed, Jul 30, 2014 at 8:52 PM, William Cox
>> <william....@distilnetworks.com> wrote:
>>>
>>> TL;DR >50% of the time I can't SSH into either my master or slave nodes
>>> and have to terminate all the machines and restart the EC2 cluster setup
>>> process.
>>>
>>> Hello,
>>>
>>> I'm trying to setup a Spark cluster on Amazon EC2. I am finding the setup
>>> script to be delicate and unpredictable in terms of reliably allowing SSH
>>> logins to all of the slaves and the master. For instance (I'm running Spark
>>> 0.9.1-hadoop1. since I intend to use Shark. I call this command to provision
>>> a 32 slave cluster using spot instances:
>>>
>>>> $./spark-ec2 --spot-price=0.1 --zone=us-east-1e -k key -i ~/key.pem -s
>>>> 32 --instance-type=m1.medium launch cluster_name
>>>
>>>
>>> After waiting for the instances to provision I get the following output:
>>>
>>>> All 32 slaves granted
>>>> Launched master in us-east-1e, regid = r-f8444a89
>>>> Waiting for instances to start up...
>>>> Waiting 120 more seconds...
>>>> Generating cluster's SSH key on master...
>>>> ssh: connect to host ecMASTER.compute-1.amazonaws.com port 22:
>>>> Connection refused
>>>> Error executing remote command, retrying after 30 seconds: Command
>>>> '['ssh', '-o', 'StrictHostKeyChecking=no', '-i', '/Users/user/key.pem',
>>>> '-t', '-t', u'r...@ecmaster.compute-1.amazonaws.com', "\n [ -f
>>>> ~/.ssh/id_rsa ] ||\n (ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa
>>>> &&\n cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys)\n "]'
>>>> returned non-zero exit status 255
>>>
>>>
>>> I have removed the key and machine names with 'MASTER' and 'key'. I wait
>>> a few more cycles of the error message and finally, after 3 attempts, the
>>> script quits with this message:
>>>
>>>
>>>> ssh: connect to host ecMASTER.compute-1.amazonaws.com port 22:
>>>> Connection refused
>>>> Error:
>>>> Failed to SSH to remote host ecMASTER.compute-1.amazonaws.com.
>>>> Please check that you have provided the correct --identity-file and
>>>> --key-pair parameters and try again.
>>>
>>> So, YES, the .pem file is correct - I am currently running a smaller
>>> cluster and can provision other machines on EC2 and use that file. Secondly,
>>> the node it can't seem to connect to is the MASTER node. I have also gone
>>> into the EC2 console and verified that all the machines are using the "key"
>>> that corresponds to "key.pem".
>>>
>>> I have tried this command 2x and on a friends machine with no success.
>>> However, I was able to provision a 15 machine cluster using m1.larges.
>>>
>>> Now I PAUSE for some period of time - 2-3 minutes (to write this email) -
>>> and I call the same command with the "--resume" flag. This time it logs into
>>> the master node just fine and begins to give the slaves SSH keys, and it
>>> fails on a certain slave.
>>>>
>>>> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22:
>>>> Connection refused
>>>> Error 255 while executing remote command, retrying after 30 seconds
>>>> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22:
>>>> Connection refused
>>>> Error 255 while executing remote command, retrying after 30 seconds
>>>> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22:
>>>> Connection refused
>>>> Error 255 while executing remote command, retrying after 30 seconds
>>>> ssh: connect to host ec2-54-237-6-95.compute-1.amazonaws.com port 22:
>>>> Connection refused
>>>> Traceback (most recent call last):
>>>> File "./spark_ec2.py", line 806, in <module>
>>>> main()
>>>> File "./spark_ec2.py", line 799, in main
>>>> real_main()
>>>> File "./spark_ec2.py", line 684, in real_main
>>>> setup_cluster(conn, master_nodes, slave_nodes, opts, True)
>>>> File "./spark_ec2.py", line 423, in setup_cluster
>>>> ssh_write(slave.public_dns_name, opts, ['tar', 'x'], dot_ssh_tar)
>>>> File "./spark_ec2.py", line 640, in ssh_write
>>>> raise RuntimeError("ssh_write failed with error %s" %
>>>> proc.returncode)
>>>> RuntimeError: ssh_write failed with error 255
>>>
>>> So I log into the EC2 console and TERMINATE that specific machine, and
>>> re-resume. Now it finally appears to be installing software on the machines.
>>>
>>> Any ideas why certain machines refuse SSH connections, or why the master
>>> refuses for several minutes and then allows?
>>>
>>> THanks.
>>>
>>> -William
>>>
>>>
>>
>
