> Andreas, to me, Twill is a library, a convenience library, that one > can use to write Yarn apps. Hoya aims to provide a general framework > using which one can take existing apps (HBase/Accumulo to start with), > and make them run well in a Yarn cluster, without intruding at all > into the App internals. The goal is to have minimal, ideally zero > changes, in the App itself. The only glue between the App and Hoya is > a Hoya interface that the App needs to implement for it to be > deployable/manageable by Hoya. > > Although, one could argue that Hoya can be written using Twill > libraries (which is something we should pursue as part of > long/medium-term collaboration between the two projects),
FWIW the first iteration was 100% groovy, using my 2012 "Grumpy" code as a starting point https://github.com/steveloughran/grumpy - The later iterations are all Java. The current design tries to have a clean model-view split server side, with all the server state stored independently from the AM itself, https://github.com/hortonworks/hoya/tree/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/state This state is something that could perhaps be worked out into something more re-usable, as all it really does is -take a specification of role types, the no. of instances of each, and some options (there's an unused anti-affinity flag) - build a model of the cluster, including live containers, queued operations, nodes in the YARN cluster and their history - Build up a list of actions: requests and releases aimed at keeping the model (and hence the deployed app) consistent with the specification. - persist placement history to HDFS for best-effort reassignment requests on cluster restart ( see https://github.com/hortonworks/hoya/blob/master/src/site/markdown/rolehistory.md ) - Respond to relayed events (container assignment, failure, rebuild after AM restart by updating model For any long-lived YARN service, the goal "keep the number of instances of a role constant" is an underlying use case, from Distributed Shell up. Hoya just effectively allows this to be driven by an JSON document persisted to HDFS, updated via RPC, and delegates to plugin providers the work of actually starting processes at the far end. I've love to see this pulled out and re-usable, indeed, I'd love to see the YARN dshell architecture redone a bit more cleanly too: https://issues.apache.org/jira/browse/YARN-1360 Not only would that give something for others to use, but there are some aspect of failure handling that we can/should improve in a way that would benefit all apps, such as better weighted moving average failure tracking, sophisticated policies on reacting to it, and moving beyond a simple works/doesn't work to a more nuanced greylist of perceived server reliability: http://steveloughran.blogspot.co.uk/2014/01/greylisting-like-blacklisting-only-more.html If we can work with the Twill team to factor this stuff into something broadly re-usable, that would be great for everyone. -steve ps. I'd a standard management service API too -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
