Hi Bharath (and rest of Spark dev list!), Just a small shout out: I am a Apache Mesos Committer and would love to help out with anything you need to get this going.
Cheers, Nik On 9 June 2015 at 21:10, Bharath Ravi Kumar <reachb...@gmail.com> wrote: > All, > > Despite the common origin of spark & mesos, the stability and adoption of > mesos, and the age of the spark-mesos binding, I find the mesos support > less mature, with fundamental shortcomings (like framework auth > <https://issues.apache.org/jira/browse/SPARK-6284>) remaining unresolved. > If there's shortage of developer time, I'd be glad to contribute, but it's > unclear if the committer group has sufficient time (and priority) to take > the mesos support forward. While it has been stated often that support for > mesos & yarn are equally important, that doesn't seem to translate to > visible progress. I'd be glad if my observation is incorrect as I seek > better focus and long term commitment on the mesos support. > As for the specific issue (6284), I'm happy to build, testing & eventually > deploy the patch in our production cluster, but I'd rather see it becoming > mainstream. > Thanks for your consideration. > > -Bharath > > > On Thu, May 28, 2015 at 9:18 AM, Bharath Ravi Kumar <reachb...@gmail.com> > wrote: > >> A follow up : considering that spark on mesos is indeed important to >> databricks, its partners and the community, fundamental issues like >> spark-6284 shouldn't be languishing for this long. A mesos cluster hosting >> diverse (i.e.multi-tenant) workloads is a common scenario in production >> for serious users. The ability to auth a framework & assign roles would be >> a fairly basic ask, one would imagine. Is the lack of time / effort a >> constraint? If so, I'd be glad to help (as mentioned in the jira). >> >> On Fri, May 15, 2015 at 5:29 PM, Iulian Dragoș < >> iulian.dra...@typesafe.com> wrote: >> >>> Hi Ankur, >>> >>> Just to add a thought to Tim's excellent answer, Spark on Mesos is very >>> important to us and is the recommended deployment for our customers as >>> Typesafe. >>> >>> Thanks for pointing to your PR, I see Tim already went through a round >>> of reviews. It seems very useful, I'll give it a try as well. >>> >>> thanks, >>> iulian >>> >>> >>> >>> On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <an...@malloc64.com> >>> wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA1 >>>> >>>> Hi Tim, >>>> >>>> Thanks for such a detailed email. I am excited to hear about the new >>>> features, I had a pull request going for adding "attribute based >>>> filtering in the mesos scheduler" but it hasn't received much love - >>>> https://github.com/apache/spark/pull/5563 . I am a fan of >>>> mesos/marathon/mesosphere and spark ecosystems and trying to push >>>> adoption at my workplace. >>>> >>>> It would love to see documentation, tutorials (anything actually) that >>>> would make mesos + spark a better and more fleshed out solution. Would >>>> it be possible for you to share some links to the JIRA and pull >>>> requests so that I can keep track on the progress/features. >>>> >>>> Again, thanks for replying. >>>> >>>> - -- Ankur Chauhan >>>> >>>> On 15/05/2015 00:39, Tim Chen wrote: >>>> > Hi Ankur, >>>> > >>>> > This is a great question as I've heard similar concerns about Spark >>>> > on Mesos. >>>> > >>>> > At the time when I started to contribute to Spark on Mesos approx >>>> > half year ago, the Mesos scheduler and related code hasn't really >>>> > got much attention from anyone and it was pretty much in >>>> > maintenance mode. >>>> > >>>> > As a Mesos PMC that is really interested in Spark I started to >>>> > refactor and check out different JIRAs and PRs around the Mesos >>>> > scheduler, and after that started to fix various bugs in Spark, >>>> > added documentation and also in fix related Mesos issues as well. >>>> > >>>> > Just recently for 1.4 we've merged in Cluster mode and Docker >>>> > support, and there are also pending PRs around framework >>>> > authentication, multi-role support, dynamic allocation, more finer >>>> > tuned coarse grain mode scheduling configurations, etc. >>>> > >>>> > And finally just want to mention that Mesosphere and Typesafe is >>>> > collaborating to bring a certified distribution >>>> > ( >>>> https://databricks.com/spark/certification/certified-spark-distributi >>>> on) >>>> > of Spark on Mesos and DCOS, and we will be pouring resources into >>>> > not just maintain Spark on Mesos but drive more features into the >>>> > Mesos scheduler and also in Mesos so stateful services can leverage >>>> > new APIs and features to make better scheduling decisions and >>>> > optimizations. >>>> > >>>> > I don't have a solidified roadmap to share yet, but we will be >>>> > discussing this and hopefully can share with the community soon. >>>> > >>>> > In summary Spark on Mesos is not dead or in maintenance mode, and >>>> > look forward to see a lot more changes from us and the community. >>>> > >>>> > Tim >>>> > >>>> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan >>>> > <an...@malloc64.com <mailto:an...@malloc64.com>> wrote: >>>> > >>>> > Hi, >>>> > >>>> > This is both a survey type as well as a roadmap query question. It >>>> > seems like of the cluster options to run spark (i.e. via YARN and >>>> > Mesos), YARN seems to be getting a lot more attention and patches >>>> > when compared to Mesos. >>>> > >>>> > Would it be correct to assume that spark on mesos is more or less >>>> > a dead or something like a maintenance-only feature and YARN is >>>> > the recommended way to go? >>>> > >>>> > What is the roadmap for spark on mesos? and what is the roadmap >>>> > for spark on yarn. I like mesos so as much as I would like to see >>>> > it thrive I don't think spark community is active (or maybe it >>>> > just appears that way). >>>> > >>>> > Another more community oriented question: what do most people use >>>> > to run spark in production or more-than-POC products? Why did you >>>> > make that decision? >>>> > >>>> > There was a similar post form early 2014 where Metei answered that >>>> > mesos and yarn were equally important, but has this changed as >>>> > spark has now reached almost 1.4.0 stage? >>>> > >>>> > -- Ankur Chauhan >>>> > >>>> > --------------------------------------------------------------------- >>>> > >>>> > >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> > <mailto:user-unsubscr...@spark.apache.org> For additional commands, >>>> > e-mail: user-h...@spark.apache.org >>>> > <mailto:user-h...@spark.apache.org> >>>> > >>>> > >>>> -----BEGIN PGP SIGNATURE----- >>>> >>>> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4 >>>> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6 >>>> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7 >>>> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8 >>>> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA >>>> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0= >>>> =4Aax >>>> -----END PGP SIGNATURE----- >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >>> >>> -- >>> >>> -- >>> Iulian Dragos >>> >>> ------ >>> Reactive Apps on the JVM >>> www.typesafe.com >>> >>> >> >
