It should be possible to improve cluster launch time if we are careful about what commands we run during setup. One way to do this would be to walk down the list of things we do for cluster initialization and see if there is anything we can do make things faster. Unfortunately this might be pretty time consuming, but I don't know of a better strategy. The place to start would be the setup.sh file at https://github.com/mesos/spark-ec2/blob/v3/setup.sh
Here are some things that take a lot of time and could be improved: 1. Creating swap partitions on all machines. We could check if there is a way to get EC2 to always mount a swap partition 2. Copying / syncing things across slaves. The copy-dir script is called too many times right now and each time it pauses for a few milliseconds between slaves [1]. This could be improved by removing unnecessary copies 3. We could make less frequently used modules like Tachyon, persistent hdfs not a part of the default setup. [1] https://github.com/mesos/spark-ec2/blob/v3/copy-dir.sh#L42 Thanks Shivaram On Sat, Jul 12, 2014 at 7:02 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > On Thu, Jul 10, 2014 at 8:10 PM, Nate D'Amico <n...@reactor8.com> wrote: > > > Starting to work through some automation/config stuff for spark stack on > > EC2 with a project, will be focusing the work through the apache bigtop > > effort to start, can then share with spark community directly as things > > progress if people are interested > > > Let us know how that goes. I'm definitely interested in hearing more. > > Nick >
