One issue with using Spark itself is that this rsync is required to get Spark to work...
Also note that a similar strategy is used for *updating* the spark cluster on ec2, where the "diff" aspect is much more important, as you might only make a small change on the driver node (recompile or reconfigure) and can get a fast sync. On Sun, May 18, 2014 at 11:22 PM, Mosharaf Chowdhury < mosharafka...@gmail.com> wrote: > What twitter calls murder, unless it has changed since then, is just a > BitTornado wrapper. In 2011, We did some comparison on the performance of > murder and the TorrentBroadcast we have right now for Spark's own broadcast > (Section 7.1 in > http://www.mosharaf.com/wp-content/uploads/orchestra-sigcomm11.pdf). > Spark's implementation was 4.5X faster than murder. > > The only issue with using TorrentBroadcast to deploy code/VM is writing a > wrapper around it to read from disk, but it shouldn't be too complicated. > If someone picks it up, I can give some pointers on how to proceed (I've > thought about doing it myself forever, but never ended up actually taking > the time; right now I don't have enough free cycles either) > > Otherwise, murder/BitTornado would be better than the current strategy we > have. > > A third option would be to use rsync; but instead of rsync-ing to every > slave from the master, one can simply rsync from the master first to one > slave; then use the two sources (master and the first slave) to rsync to > two more; then four and so on. Might be a simpler solution without many > changes. > > -- > Mosharaf Chowdhury > http://www.mosharaf.com/ > > > On Sun, May 18, 2014 at 11:07 PM, Andrew Ash <and...@andrewash.com> wrote: > >> My first thought would be to use libtorrent for this setup, and it turns >> out that both Twitter and Facebook do code deploys with a bittorrent setup. >> Twitter even released their code as open source: >> >> >> https://blog.twitter.com/2010/murder-fast-datacenter-code-deploys-using-bittorrent >> >> >> http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/ >> >> >> On Sun, May 18, 2014 at 10:44 PM, Daniel Mahler <dmah...@gmail.com>wrote: >> >>> I am not an expert in this space either. I thought the initial rsync >>> during launch is really just a straight copy that did not need the tree >>> diff. So it seemed like having the slaves do the copying among it each >>> other would be better than having the master copy to everyone directly. >>> That made me think of bittorrent, though there may well be other systems >>> that do this. >>> From the launches I did today it seems that it is taking around 1 minute >>> per slave to launch a cluster, which can be a problem for clusters with 10s >>> or 100s of slaves, particularly since on ec2 that time has to be paid for. >>> >>> >>> On Sun, May 18, 2014 at 11:54 PM, Aaron Davidson <ilike...@gmail.com>wrote: >>> >>>> Out of curiosity, do you have a library in mind that would make it easy >>>> to setup a bit torrent network and distribute files in an rsync (i.e., >>>> apply a diff to a tree, ideally) fashion? I'm not familiar with this space, >>>> but we do want to minimize the complexity of our standard ec2 launch >>>> scripts to reduce the chance of something breaking. >>>> >>>> >>>> On Sun, May 18, 2014 at 9:22 PM, Daniel Mahler <dmah...@gmail.com>wrote: >>>> >>>>> I am launching a rather large cluster on ec2. >>>>> It seems like the launch is taking forever on >>>>> .... >>>>> Setting up spark >>>>> RSYNC'ing /root/spark to slaves... >>>>> ... >>>>> >>>>> It seems that bittorrent might be a faster way to replicate >>>>> the sizeable spark directory to the slaves >>>>> particularly if there is a lot of not very powerful slaves. >>>>> >>>>> Just a thought ... >>>>> >>>>> cheers >>>>> Daniel >>>>> >>>>> >>>> >>> >> >
