On 12 April 2014 23:38, Roman Shaposhnik <r...@apache.org> wrote: > On Sat, Apr 12, 2014 at 11:58 AM, Andrew Purtell <apurt...@apache.org> > wrote: > >> > The reason I ask is I'm wondering how Slider differentiates from > projects > >> > like Apache Twill or Apache Bigtop that are already existing vehicles > for > >> > achieving the aims discussed in the Slider proposal. > >> > >> > >> Twill: handles all the AM logic for running new code packaged as a JAR > with > >> an executor method > >> Bigtop: stack testing > > > > > > As a Bigtop committer, I disagree with this narrow interpretation of the > > scope of the project, but this is my personal opinion and I am not PMC... > > A strong +1 here! Bigtop attacks a problem of packaging and deploying > Hadoop stacks from a classical UNIX packaging background.
I think I'm a big top committer too --but sorry if I stepped on your toes. I view it as testing primarily because the main bits of code I've been near were stuff to do FS testing, and the script testing code ( we use org.apache.bigtop.itest.shell.Shell as the basis for our remote functional tests). I'd also like to get > We are > also slowly moving into container/VM/OSv packaging territory which > could be an extremely exciting way of side-stepping the general installer > issues (something that Ambari struggles mightily with). > Topic for another day. I believe I have some past experience in that area, hence some opinions. > Something like Slider tries to leapfrog and side-step UNIX packaging and > deployment altogether. This is an interesting take on the problem, but > ultimately the jury is still out on what the level of adoption for > "everything > is now YARN" will be. > What it's trying to do is say "end users are allowed to install and run things too". It's like in linux, you can do rpm/deb installations -and that's the best way to manage a set of machines, but there's nothing to stop me, a non-root user, from running anything I want even if the admins haven't installed it. We have formats for that .tar, .gz, arguably .war and .ear. most of the standard java server apps come with .tar.gz/.zip distributions (hadoop included), so we all know user-side code is a use case -what we're thinking of here are : what are the problems we need to address to get this to work in a YARN cluster. -getting the configuration customised to the destination. This is the perennial problem of Configuration Management -and I am not going to try and solve that problem -again. There is 20+ years of history there, the scripted vs desired state, centralised vs decentralised and static/dynamic debates being the perennial ones, ones into which you can device them all up. Oh, and there's policy driven, such as CompSolv [Hewson12], which is not yet something that's trickled into the OSS tools yet. What we're thinking of is something far less ambitious than "lets do a new CM system" , and in doing so have something more practical "how to make it easy to bundle non-YARN code so that YARN can deploy it, with enough metadata and scripts to bring it up. At the same, I do want it to be possible for CM systems, PaaS front ends and other management tools to be able to not just spawn a slider app, but integrate better with it. Part of our thinking here is the idea of "bonded" AM, which, rather than an in-process "unmanaged" AM, is a remote YARN-deployed AM but hooked up so that only the owner process can work with it -events being published for the owner to track and control it, and having both slider and the owner being able to tell any agent what to do. This is work in progress -I'd like a prototype web front end though. We are going to have to do some of the dynamic config generation, because we can't dictate up front ports and hosts, so will need to get that information into both the configs used when starting the apps, and into information published up for clients to pull down -the latter meaning we are going to have to evolve a cluster registry, based on Curator or something else -adding artifact sharing/generation too. A registry is likely to go beyond slder, as it shouldn't be restricted to just one YARN app, and indeed, there's no reason why, say, a bigtop deployed HBase cluster can't publish its binding information. YARN-913 <https://issues.apache.org/jira/browse/YARN-913> has raised some of this; I'd like to use slider to evolve some of the topics with a faster cycle than Hadoop releases -but I'd also love to have any other YARN apps involved (given I'm trying to re-use any existing code that works: Apache Curator, Apache Directory, hopefully bits of Helix, I'd welcome other contribs too). > > At the end of the day, we will need both for a really long time. > +1. YARN itself depends on a well-setup cluster, network and all the bits where users take an ops team for granted > > > For example, we package Hadoop core and ecosystem services both for > > deployment, have Puppet based deployment automation (which can be used > more > > generally than merely for setting up test clusters), and I have been > > considering filing JIRAs to tie in cgroups at the whole stack level here. > > > What is missing of course is a hierarchical model for resource > management, > > and tools within the components for differentiated service levels, but > that > > is another discussion. > > On that note, I find YARN's attitude towards cgroups be, how shall I put > it, > optimistic. If you look carefully you can see that the Linux community > has completely given up an pretending that one can use naked cgroup > trees for reliable resource partitioning: > http://www.freedesktop.org/wiki/Software/systemd/PaxControlGroups/ > > It is now clear to me that the path Linux distros are endorsing is via > the brokers such as systemd: > > http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/ > > As such I'd see Bigtop providing quite a bit of value out of the box > via tight integration with systemd. YARN, at the moment, is in a trickier > situation. > I'd take that up in the YARN list in general -its only just moving from memory limiting to CPU, then we can worry about what is possible with IO and network. Which may mean it is still time to impose a cgroup architecture that all are happy with. Of course, in a VM world, resource throttling can take place outside the OS, and on a cluster-wide level. Things get -possibly- simpler. You can stop worrying about cleanup too. -Steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
