Hi Ankur, Just to add a thought to Tim's excellent answer, Spark on Mesos is very important to us and is the recommended deployment for our customers as Typesafe.
Thanks for pointing to your PR, I see Tim already went through a round of reviews. It seems very useful, I'll give it a try as well. thanks, iulian On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <[email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Tim, > > Thanks for such a detailed email. I am excited to hear about the new > features, I had a pull request going for adding "attribute based > filtering in the mesos scheduler" but it hasn't received much love - > https://github.com/apache/spark/pull/5563 . I am a fan of > mesos/marathon/mesosphere and spark ecosystems and trying to push > adoption at my workplace. > > It would love to see documentation, tutorials (anything actually) that > would make mesos + spark a better and more fleshed out solution. Would > it be possible for you to share some links to the JIRA and pull > requests so that I can keep track on the progress/features. > > Again, thanks for replying. > > - -- Ankur Chauhan > > On 15/05/2015 00:39, Tim Chen wrote: > > Hi Ankur, > > > > This is a great question as I've heard similar concerns about Spark > > on Mesos. > > > > At the time when I started to contribute to Spark on Mesos approx > > half year ago, the Mesos scheduler and related code hasn't really > > got much attention from anyone and it was pretty much in > > maintenance mode. > > > > As a Mesos PMC that is really interested in Spark I started to > > refactor and check out different JIRAs and PRs around the Mesos > > scheduler, and after that started to fix various bugs in Spark, > > added documentation and also in fix related Mesos issues as well. > > > > Just recently for 1.4 we've merged in Cluster mode and Docker > > support, and there are also pending PRs around framework > > authentication, multi-role support, dynamic allocation, more finer > > tuned coarse grain mode scheduling configurations, etc. > > > > And finally just want to mention that Mesosphere and Typesafe is > > collaborating to bring a certified distribution > > (https://databricks.com/spark/certification/certified-spark-distributi > on) > > of Spark on Mesos and DCOS, and we will be pouring resources into > > not just maintain Spark on Mesos but drive more features into the > > Mesos scheduler and also in Mesos so stateful services can leverage > > new APIs and features to make better scheduling decisions and > > optimizations. > > > > I don't have a solidified roadmap to share yet, but we will be > > discussing this and hopefully can share with the community soon. > > > > In summary Spark on Mesos is not dead or in maintenance mode, and > > look forward to see a lot more changes from us and the community. > > > > Tim > > > > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan > > <[email protected] <mailto:[email protected]>> wrote: > > > > Hi, > > > > This is both a survey type as well as a roadmap query question. It > > seems like of the cluster options to run spark (i.e. via YARN and > > Mesos), YARN seems to be getting a lot more attention and patches > > when compared to Mesos. > > > > Would it be correct to assume that spark on mesos is more or less > > a dead or something like a maintenance-only feature and YARN is > > the recommended way to go? > > > > What is the roadmap for spark on mesos? and what is the roadmap > > for spark on yarn. I like mesos so as much as I would like to see > > it thrive I don't think spark community is active (or maybe it > > just appears that way). > > > > Another more community oriented question: what do most people use > > to run spark in production or more-than-POC products? Why did you > > make that decision? > > > > There was a similar post form early 2014 where Metei answered that > > mesos and yarn were equally important, but has this changed as > > spark has now reached almost 1.4.0 stage? > > > > -- Ankur Chauhan > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > <mailto:[email protected]> For additional commands, > > e-mail: [email protected] > > <mailto:[email protected]> > > > > > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4 > xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6 > PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7 > A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8 > tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA > w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0= > =4Aax > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com
