hi, thanks for your answer!
I have few more: 1) the file /root/spark/conf/slaves , has the full DNS names of servers ( ec2-52-26-7-137.us-west-2.compute.amazonaws.com), did you add there the internal ip? 2) You call to start-all. Isn't it too aggressive? Let's say I have 20 slaves up, and I want to add one more, why should we stop the entire cluster for this? thanks, nizan On Thu, May 28, 2015 at 10:19 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > I do this way: > > - Launch a new instance by clicking on the slave instance and choose *launch > more like this * > *- *Once its launched, ssh into it and add the master public key to > .ssh/authorized_keys > - Add the slaves internal IP to the master's conf/slaves file > - do sbin/start-all.sh and it will show up along with other slaves. > > > > Thanks > Best Regards > > On Thu, May 28, 2015 at 12:29 PM, nizang <ni...@windward.eu> wrote: > >> hi, >> >> I'm working on spark standalone system on ec2, and I'm having problems on >> resizing the cluster (meaning - adding or removing slaves). >> >> In the basic ec2 scripts >> (http://spark.apache.org/docs/latest/ec2-scripts.html), there's only >> script >> for lunching the cluster, not adding slaves to it. On the spark-standalone >> page >> ( >> http://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts >> ), >> I can see only options for stopping and starting slaves, not adding them. >> >> What I try to do now (as a bad workaround...), is the following: >> >> 1) Go to the ec2 UI, create image from the current slave >> 2) Lunch new instance based on this image >> 3) Copy the public DNS of this slave >> 4) SSH to the master, and edit the file >> "/root/spark-ec2/ec2-variables.sh", >> and add the DNS to the "export SLAVES" variables >> 5) Running the script /root/spark-ec2/setup.sh >> >> After doing the above steps, I can see the new slave in the UI (8080 port) >> of the master. However, this solution is bad for many reasons: >> >> 1) It requires many manual steps >> 2) It requires stopping and starting the cluster >> 3) There's no auto-detection in case slave stopped >> >> and many other reasons... >> >> oes anybody have another idea on how to add/remove slaves for standalone >> on >> a simple and safe way? >> >> thanks, nizan >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Adding-slaves-on-spark-standalone-on-ec2-tp23064.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
