+1.. Very interesting stuff.. thanks, Thilina
On Sun, Jun 26, 2011 at 7:12 PM, Suresh Marru <sma...@apache.org> wrote: > Interesting Project. Time permitting, I would like to contribute to the > workflow effort > > --Suresh > > On Jun 24, 2011, at 3:46 PM, Mohammad Islam wrote: > > > Hi, > > > > I would like to propose Oozie to be an Apache Incubator project. > > Oozie is a server-based workflow scheduling and coordination system to > manage > > data processing jobs for Apache Hadoop. > > > > > > Here's a link to the proposal in the Incubator wiki > > http://wiki.apache.org/incubator/OozieProposal > > > > > > I've also pasted the initial contents below. > > > > Regards, > > > > Mohammad Islam > > > > > > Start of Oozie Proposal > > > > Abstract > > Oozie is a server-based workflow scheduling and coordination system to > manage > > data processing jobs for Apache HadoopTM. > > > > Proposal > > Oozie is an extensible, scalable and reliable system to define, manage, > > schedule, and execute complex Hadoop workloads via web services. More > > specifically, this includes: > > > > * XML-based declarative framework to specify a job or a complex > workflow of > > dependent jobs. > > > > * Support different types of job such as Hadoop Map-Reduce, Pipe, > Streaming, > > Pig, Hive and custom java applications. > > > > * Workflow scheduling based on frequency and/or data availability. > > * Monitoring capability, automatic retry and failure handing of > jobs. > > * Extensible and pluggable architecture to allow arbitrary grid > programming > > paradigms. > > > > * Authentication, authorization, and capacity-aware load throttling > to allow > > multi-tenant software as a service. > > > > Background > > Most data processing applications require multiple jobs to achieve their > goals, > > with inherent dependencies among the jobs. A dependency could be > sequential, > > where one job can only start after another job has finished. Or it could > be > > conditional, where the execution of a job depends on the return value or > status > > of another job. In other cases, parallel execution of multiple jobs may > be > > permitted – or desired – to exploit the massive pool of compute nodes > provided > > by Hadoop. > > > > These job dependencies are often expressed as a Directed Acyclic Graph, > also > > called a workflow. A node in the workflow is typically a job (a > computation on > > the grid) or another type of action such as an eMail notification. > Computations > > can be expressed in map/reduce, Pig, Hive or any other programming > paradigm > > available on the grid. Edges of the graph represent transitions from one > node > > to the next, as the execution of a workflow proceeds. > > > > Describing a workflow in a declarative way has the advantage of > decoupling job > > dependencies and execution control from application logic. Furthermore, > the > > workflow is modularized into jobs that can be reused within the same > workflow > > or across different workflows. Execution of the workflow is then driven > by a > > runtime system without understanding the application logic of the jobs. > This > > runtime system specializes in reliable and predictable execution: It can > retry > > actions that have failed or invoke a cleanup action after termination of > the > > workflow; it can monitor progress, success, or failure of a workflow, > and send > > appropriate alerts to an administrator. The application developer is > relieved > > from implementing these generic procedures. > > > > Furthermore, some applications or workflows need to run in periodic > intervals > > or when dependent data is available. For example, a workflow could be > executed > > every day as soon as output data from the previous 24 instances of > another, > > hourly workflow is available. The workflow coordinator provides such > scheduling > > features, along with prioritization, load balancing and throttling to > optimize > > utilization of resources in the cluster. This makes it easier to > maintain, > > control, and coordinate complex data applications. > > > > Nearly three years ago, a team of Yahoo! developers addressed these > critical > > requirements for Hadoop-based data processing systems by developing a > new > > workflow management and scheduling system called Oozie. While it was > initially > > developed as a Yahoo!-internal project, it was designed and implemented > with > > the intention of open-sourcing. Oozie was released as a GitHub project in > early > > 2010. Oozie is used in production within Yahoo and since it has been > > open-sourced it has been gaining adoption with external developers > > > > Rationale > > Commonly, applications that run on Hadoop require multiple Hadoop jobs > in order > > to obtain the desired results. Furthermore, these Hadoop jobs are > commonly a > > combination of Java map-reduce jobs, Streaming map-reduce jobs, Pipes > > map-reduce jobs, Pig jobs, Hive jobs, HDFS operations, Java programs and > shell > > scripts. > > > > Because of this, developers find themselves writing ad-hoc glue programs > to > > combine these Hadoop jobs. These ad-hoc programs are difficult to > schedule, > > manage, monitor and recover. > > > > Workflow management and scheduling is an essential feature for > large-scale data > > processing applications. Such applications could write the customized > solution > > that would require separate development, operational, and maintenance > overhead. > > Since it is a prevalent use-case for data processing, the application > developer > > would surely prefer a generalized solution with little or no such > overhead. > > Oozie addresses the challenge by providing an execution framework to > flexibly > > specify the job dependency, data dependency, and time dependency. In > addition, > > Oozie provides a multi-tenant-based centralized service and the > opportunity to > > optimize load and utilization while respecting SLAs. > > > > Oozie is built on Apache Hadoop to schedule jobs related to various > Apache > > projects such as Hadoop, Pig, and Hive. As an Apache Open source > project, Oozie > > is expected to attract the larger and more diversified community that > currently > > uses such Apache sponsored projects. Additionally, users of the Hadoop > > ecosystem can influence Oozie’s roadmap, and contribute to it. Likewise, > Oozie, > > as part of the Apache Hadoop ecosystem, will be a great benefit to the > current > > Hadoop/Pig/Hive/HBase/HCatalog community. > > > > Current Status > > Meritocracy > > Oozie currently is a github-based open sourced project where developers > from > > multiple companies are contributing to the project. Our intent with this > > incubator proposal is to further extend this diverse developer community > around > > Oozie following the Apache meritocracy model. We plan to continue to > provide > > adequate support to new developers and to quickly recruit those who make > solid > > contributions to committer status. In addition, Oozie will expect, > accept, and > > work to attract contributions from amateurs as well. > > > > Community > > While an efficient workflow management and scheduling system is critical > for > > large companies with huge data processing in multi-tenant clusters, it > is > > equally necessary for any non-trivial deployment. Different companies > are > > currently using Oozie as a workflow scheduler for Hadoop-based data > processing. > > At Yahoo! it is being used extensively in production clusters to process > > thousand of jobs. Like the Oozie user community, the Oozie developer > community > > is also very strong. Developers from Yahoo! provided the initial code > base, and > > they are still the most active contributors. In late 2010, developers > from > > Cloudera also started contributing, and currently other companies (e.g., > IBM) > > are beginning to participate. > > > > We currently use JIRA for issue tracking, github for code hosting and > Yahoo! > > Groups for developer and user communications. > > > > Core Developers > > Oozie is currently being designed and developed by four engineers from > Yahoo! – > > Mohammad Islam, Angelo Huang, Mayank Bansal, and Andreas Neumann. In > addition, > > many outside contributors are actively contributing in design and > development. > > Among them, Alejandro Abdelnur from Cloudera and Chao Wang from IBM are > very > > important contributors. All of these core developers have deep expertise > in > > Hadoop and the Hadoop Ecosystem in general. > > > > Alignment > > The ASF is a natural host for Oozie given that it is already the home of > > Hadoop, Pig, Hive, and other emerging cloud software projects. Oozie was > > designed to support Hadoop from the beginning in order to solve data > processing > > challenges in Hadoop clusters. Oozie complements the existing Apache > cloud > > computing projects by providing a flexible framework for managing > complex data > > processing tasks. > > > > Known Risks > > Orphaned Products > > The core developers plan to work full time on the project. There is very > little > > risk of Oozie getting orphaned since large companies like Yahoo! are > > extensively using it on their production Hadoop clusters. For example, > there > > are nearly 400 Yahoo! internal Oozie users and thousands of jobs are > processed > > hourly through Oozie in production. In addition, there are nearly 400 > active > > users (including Yahoo! internal and external) in the email community > where > > nearly 15 emails are exchanged per day. Furthermore, there were more > than 1500 > > downloads of the Oozie binary in the last eight months from the github > site and > > a large number of downloads were conducted by other companies such as > Cloudera. > > Oozie has three major releases and more than 15 patch releases in the > last > > couple of years which further demonstrates Oozie as a very active > project. We > > plan to extend and diversify this community further through Apache. > > > > Inexperience with Open Source > > The core developers are all active users and followers of open source. > They are > > already committers and contributors to the Oozie Github project. In > addition, > > they are very familiar with Apache principals and philosophy for > community > > driven software development. > > > > Homogeneous Developers > > The core developers are from Yahoo! as well as from several other > corporations, > > including Cloudera and IBM. > > > > Reliance on Salaried Developers > > Currently, the developers are paid to do work on Oozie. Companies like > Yahoo! > > and Cloudera are invested in Oozie as the solution to the workflow > management > > and scheduling problem in Hadoop clusters, and that is not likely to > change. In > > addition, since workflow management is very important for most hadoop > based > > data processing, non-salaried developers and researchers from various > > institutes are expected to contribute to the project. > > > > Relationships with Other Apache Products > > Oozie is based on Apache Hadoop to manage jobs created by different > Apache > > projects such as Hadoop, Pig, and Hive. Users of these products are > extensively > > using Oozie as their workflow scheduler. > > > > An Excessive Fascination with the Apache Brand > > We deeply respect the reputation of Apache and have had great success > with > > other Apache projects such as Pig and HCatalog. We are motivated to > expand and > > increase the adoption and development of Oozie following Apache’s > established > > open source model. We have also given reasons in the Rationale and > Alignment > > sections. > > > > Documentation > > Information about Oozie can be found at http://yahoo.github.com/oozie/. > The > > following links provide more information about Oozie in open source: > > > > * Codebase at GitHub: https://github.com/yahoo/oozie. > > * JIRA : http://oozie-jira.hadoop.developer.yahoo.net > > * Continuous Integration (CI) build: > > http://oozie-ci.hadoop.developer.yahoo.net/ > > > > * Yahoo user community: > http://tech.groups.yahoo.com/group/Oozie-users/ > > Initial Source > > Oozie has been under development since 2009 by a team of engineers at > Yahoo!. It > > is currently hosted on GitHub under an Apache license at > > https://github.com/yahoo/oozie. > > > > External Dependencies > > The required external dependencies are all Apache License or compatible > > licenses. Following the components with non-Apache licenses are > enumerated: > > > > * HSQLDB License: HSQLDB > > * JDOM license: JDOM > > * BSD: Serp > > * CCDL v1: jaxb-api, ejb, JAF > > NOTE: With the exception of HSQLDB and JDOM that are directly used by > Oozie, > > the other listed components are transitive dependencies of other Apache > > components used by Oozie. > > > > Cryptography > > Oozie supports the Kerberos authentication mechanism to access secured > Hadoop > > services. > > > > Required Resources > > Mailing Lists > > * oozie-private for private PMC discussions (with moderated > subscriptions) > > * oozie-dev > > * oozie-commits > > * oozie-user > > Subversion Directory > > https://svn.apache.org/repos/asf/incubator/oozie > > Issue Tracking > > JIRA Oozie (OOZIE) > > Other Resources > > The existing code already has unit tests, so we would like a Hudson > instance > > to run them whenever a new patch is submitted. This can be added after > project > > creation. > > > > Initial Committers > > * Mohammad K Islam (mislam77 at yahoo dot com) > > * Angelo K Huang (angelohuang at gmail dot com) > > * Mayank Bansal (mabansal at gmail dot com) > > * Andreas Neumann (neunand at gmail dot com) > > * Alejandro Abdelnur (tucu00 at gmail dot com) > > * Chao Wang (brookwc at gmail dot com) > > Affiliations > > * Mohammad K Islam (Yahoo!) > > * Angelo Huang (Yahoo!) > > * Mayank Bansal (Yahoo!) > > * Andreas Neumann (Yahoo!) > > * Alejandro Abdelnur (Cloudera) > > * Chao Wang (IBM) > > Sponsors > > Champion > > Alan Gates > > Nominated Mentors > > * Owen O'Malley (Incubator PMC member) > > * Alan Gates (Incubator PMC member) > > * Christopher Douglas(Incubator PMC member) > > * Devaraj Das (Hadoop PMC member) > > Sponsoring EntityWe are requesting the Incubator to sponsor this project. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- https://www.cs.indiana.edu/~tgunarat/ http://www.linkedin.com/in/thilina http://thilina.gunarathne.org
