A follow up : considering that spark on mesos is indeed important to databricks, its partners and the community, fundamental issues like spark-6284 shouldn't be languishing for this long. A mesos cluster hosting diverse (i.e.multi-tenant) workloads is a common scenario in production for serious users. The ability to auth a framework & assign roles would be a fairly basic ask, one would imagine. Is the lack of time / effort a constraint? If so, I'd be glad to help (as mentioned in the jira).
On Fri, May 15, 2015 at 5:29 PM, Iulian Dragoș <iulian.dra...@typesafe.com> wrote: > Hi Ankur, > > Just to add a thought to Tim's excellent answer, Spark on Mesos is very > important to us and is the recommended deployment for our customers as > Typesafe. > > Thanks for pointing to your PR, I see Tim already went through a round of > reviews. It seems very useful, I'll give it a try as well. > > thanks, > iulian > > > > On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Hi Tim, >> >> Thanks for such a detailed email. I am excited to hear about the new >> features, I had a pull request going for adding "attribute based >> filtering in the mesos scheduler" but it hasn't received much love - >> https://github.com/apache/spark/pull/5563 . I am a fan of >> mesos/marathon/mesosphere and spark ecosystems and trying to push >> adoption at my workplace. >> >> It would love to see documentation, tutorials (anything actually) that >> would make mesos + spark a better and more fleshed out solution. Would >> it be possible for you to share some links to the JIRA and pull >> requests so that I can keep track on the progress/features. >> >> Again, thanks for replying. >> >> - -- Ankur Chauhan >> >> On 15/05/2015 00:39, Tim Chen wrote: >> > Hi Ankur, >> > >> > This is a great question as I've heard similar concerns about Spark >> > on Mesos. >> > >> > At the time when I started to contribute to Spark on Mesos approx >> > half year ago, the Mesos scheduler and related code hasn't really >> > got much attention from anyone and it was pretty much in >> > maintenance mode. >> > >> > As a Mesos PMC that is really interested in Spark I started to >> > refactor and check out different JIRAs and PRs around the Mesos >> > scheduler, and after that started to fix various bugs in Spark, >> > added documentation and also in fix related Mesos issues as well. >> > >> > Just recently for 1.4 we've merged in Cluster mode and Docker >> > support, and there are also pending PRs around framework >> > authentication, multi-role support, dynamic allocation, more finer >> > tuned coarse grain mode scheduling configurations, etc. >> > >> > And finally just want to mention that Mesosphere and Typesafe is >> > collaborating to bring a certified distribution >> > (https://databricks.com/spark/certification/certified-spark-distributi >> on) >> > of Spark on Mesos and DCOS, and we will be pouring resources into >> > not just maintain Spark on Mesos but drive more features into the >> > Mesos scheduler and also in Mesos so stateful services can leverage >> > new APIs and features to make better scheduling decisions and >> > optimizations. >> > >> > I don't have a solidified roadmap to share yet, but we will be >> > discussing this and hopefully can share with the community soon. >> > >> > In summary Spark on Mesos is not dead or in maintenance mode, and >> > look forward to see a lot more changes from us and the community. >> > >> > Tim >> > >> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan >> > <an...@malloc64.com <mailto:an...@malloc64.com>> wrote: >> > >> > Hi, >> > >> > This is both a survey type as well as a roadmap query question. It >> > seems like of the cluster options to run spark (i.e. via YARN and >> > Mesos), YARN seems to be getting a lot more attention and patches >> > when compared to Mesos. >> > >> > Would it be correct to assume that spark on mesos is more or less >> > a dead or something like a maintenance-only feature and YARN is >> > the recommended way to go? >> > >> > What is the roadmap for spark on mesos? and what is the roadmap >> > for spark on yarn. I like mesos so as much as I would like to see >> > it thrive I don't think spark community is active (or maybe it >> > just appears that way). >> > >> > Another more community oriented question: what do most people use >> > to run spark in production or more-than-POC products? Why did you >> > make that decision? >> > >> > There was a similar post form early 2014 where Metei answered that >> > mesos and yarn were equally important, but has this changed as >> > spark has now reached almost 1.4.0 stage? >> > >> > -- Ankur Chauhan >> > >> > --------------------------------------------------------------------- >> > >> > >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > <mailto:user-unsubscr...@spark.apache.org> For additional commands, >> > e-mail: user-h...@spark.apache.org >> > <mailto:user-h...@spark.apache.org> >> > >> > >> -----BEGIN PGP SIGNATURE----- >> >> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4 >> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6 >> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7 >> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8 >> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA >> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0= >> =4Aax >> -----END PGP SIGNATURE----- >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > > -- > > -- > Iulian Dragos > > ------ > Reactive Apps on the JVM > www.typesafe.com > >
