Please add the proposal to the official incubator proposal wiki list https://wiki.apache.org/incubator/ProjectProposals
Craig > On Feb 4, 2018, at 1:10 PM, Byung-Gon Chun <bgc...@gmail.com> wrote: > > Hi, > > 72 hours has passed and the vote for accepting Coral into the Apache > Incubator has passed with: > > 9 binding "+1" votes, 1 non-binding "+1" votes, and no "-1” votes. > > Binding votes: > Kevin A. McGrail > Davor Bonaci > Dave Fisher > Hyunsik Choi > Leif Hedstrom > Jean-Baptiste Onofré > Romain Manni-Bucau > Mark Struberg > Byung-Gon Chun > > Non-binding votes: > Clebert Suconic > > Thanks to everyone who voted. > > On Thu, Feb 1, 2018 at 11:07 PM, Byung-Gon Chun <bgc...@gmail.com> wrote: > >> Hi all, >> >> I would like to start a VOTE to propose the Coral project as a podling >> into the Apache Incubator. >> >> The ASF voting rules are described at https://www.apache.org/foundation/ >> voting.html >> >> A vote for accepting a new Apache Incubator podling is a majority vote for >> which only Incubator PMC member votes are binding. >> >> This vote will run for at least 72 hours. Please VOTE as follows. >> [] +1 Accept Coral into the Apache Incubator >> [] +0 Abstain >> [] -1 Do not accept Coral into the Apache Incubator because ... >> >> The proposal is listed below, but you can also access it on the wiki: >> https://wiki.apache.org/incubator/CoralProposal >> >> = CoralProposal = >> >> == Abstract == >> Coral is a data processing system for flexible employment with different >> execution scenarios for various deployment characteristics on clusters. >> >> == Proposal == >> Today, there is a wide variety of data processing systems with different >> designs for better performance and datacenter efficiency. They include >> processing data on specific resource environments and running jobs with >> specific attributes. Although each system successfully solves the problems >> it targets, most systems are designed in the way that runtime behaviors are >> built tightly inside the system core to hide the complexity of distributed >> computing. This makes it hard for a single system to support different >> deployment characteristics with different runtime behaviors without >> substantial effort. >> >> Coral is a data processing system that aims to flexibly control the runtime >> behaviors of a job to adapt to varying deployment characteristics. Moreover, >> it provides a means of extending the system’s capabilities and incorporating >> the extensions to the flexible job execution. >> >> In order to be able to easily modify runtime behaviors to adapt to varying >> deployment characteristics, Coral exposes runtime behaviors to be flexibly >> configured and modified at both compile-time and runtime through a set of >> high-level graph pass interfaces. >> >> We hope to contribute to the big data processing community by enabling more >> flexibility and extensibility in job executions. Furthermore, we can benefit >> more together as a community when we work together as a community to mature >> the system with more use cases and understanding of diverse deployment >> characteristics. The Apache Software Foundation is the perfect place to >> achieve these aspirations. >> >> == Background == >> Many data processing systems have distinctive runtime behaviors optimized >> and configured for specific deployment characteristics like different >> resource environments and for handling special job attributes. >> >> For example, much research have been conducted to overcome the challenge of >> running data processing jobs on cheap, unreliable transient resources. >> Likewise, techniques for disaggregating different types of resources, like >> memory, CPU and GPU, are being actively developed to use datacenter >> resources more efficiently. Many researchers are also working to run data >> processing jobs in even more diverse environments, such as across distant >> datacenters. Similarly, for special job attributes, many works take >> different approaches, such as runtime optimization, to solve problems like >> data skew, and to optimize systems for data processing jobs with small-scale >> input data. >> >> Although each of the systems performs well with the jobs and in the >> environments they target, they perform poorly with unconsidered cases, and >> do not consider supporting multiple deployment characteristics on a single >> system in their designs. >> >> For an application writer to optimize an application to perform well on a >> certain system engraved with its underlying behaviors, it requires a deep >> understanding of the system itself, which is an overhead that often requires >> a lot of time and effort. Moreover, for a developer to modify such system >> behaviors, it requires modifications of the system core, which requires an >> even deeper understanding of the system itself. >> >> With this background, Coral is designed to represent all of its jobs as an >> Intermediate Representation (IR) DAG. In the Coral compiler, user >> applications from various programming models (ex. Apache Beam) are >> submitted, transformed to an IR DAG, and optimized/customized for the >> deployment characteristics. In the IR DAG optimization phase, the DAG is >> modified through a series of compiler “passes” which reshape or annotate the >> DAG with an expression of the underlying runtime behaviors. The IR DAG is >> then submitted as an execution plan for the Coral runtime. The runtime >> includes the unmodified parts of data processing in the backbone which is >> transparently integrated with configurable components exposed for further >> extension. >> >> == Rationale == >> Coral’s vision lies in providing means for flexibly supporting a wide >> variety of job execution scenarios for users while facilitating system >> developers to extend the execution framework with various functionalities at >> the same time. The capabilities of the system can be extended as it grows to >> meet a more variety of execution scenarios. We require inputs from users and >> developers from diverse domains in order to make it a more thriving and >> useful project. The Apache Software Foundation provides the best tools and >> community to support this vision. >> >> == Initial Goals == >> Initial goals will be to move the existing codebase to Apache and integrate >> with the Apache development process. We further plan to develop our system >> to meet the needs for more execution scenarios for a more variety of >> deployment characteristics. >> >> == Current Status == >> Coral codebase is currently hosted in a repository at github.com. The >> current version has been developed by system developers at Seoul National >> University, Viva Republica, Samsung, and LG. >> >> == Meritocracy == >> We plan to strongly support meritocracy. We will discuss the requirements in >> an open forum, and those that continuously contribute to Coral with the >> passion to strengthen the system will be invited as committers. Contributors >> that enrich Coral by providing various use cases, various implementations of >> the configurable components including ideas for optimization techniques will >> be especially welcome. Committers with a deep understanding of the system’s >> technical aspects as a whole and its philosophy will definitely be voted as >> the PMC. We will monitor community participation so that privileges can be >> extended to those that contribute. >> >> == Community == >> We hope to expand our contribution community by becoming an Apache incubator >> project. The contributions will come from both users and system developers >> interested in flexibility and extensibility of job executions that Coral can >> support. We expect users to mainly contribute to diversify the use cases and >> deployment characteristics, and developers to contribute to implement them. >> >> == Alignment == >> Apache Spark is one of many popular data processing frameworks. The system >> is designed towards optimizing jobs using RDDs in memory and many other >> optimizations built tightly within the framework. In contrast to Spark, >> Coral aims to provide more flexibility for job execution in an easy manner. >> >> Apache Tez enables developers to build complex task DAGs with control over >> the control plane of job execution. In Coral, a high-level programming layer >> (ex. Apache Beam) is automatically converted to a basic IR DAG and can be >> converted to any IR DAG through a series of easy user writable passes, that >> can both reshape and modify the annotation (of execution properties) of the >> DAG. Moreover, Coral leaves more parts of the job execution configurable, >> such as the scheduler and the data plane. As opposed to providing a set of >> properties for solid optimization, Coral’s configurable parts can be easily >> extended and explored by implementing the pre-defined interfaces. For >> example, an arbitrary intermediate data store can be added. >> >> Coral currently supports Apache Beam programs and we are working on >> supporting Apache Spark programs as well. Coral also utilizes Apache REEF >> for container management, which allows Coral to run in Apache YARN and >> Apache Mesos clusters. If necessary, we plan to contribute to and >> collaborate with these other Apache projects for the benefit of all. We plan >> to extend such integrations with more Apache softwares. Apache software >> foundation already hosts many major big-data systems, and we expect to help >> further growth of the big-data community by having Coral within the Apache >> foundation. >> >> == Known Risks == >> === Orphaned Products === >> The risk of the Coral project being orphaned is minimal. There is already >> plenty of work that arduously support different deployment characteristics, >> and we propose a general way to implement them with flexible and extensible >> configuration knobs. The domain of data processing is already of high >> interest, and this domain is expected to evolve continuously with various >> other purposes, such as resource disaggregation and using transient >> resources for better datacenter resource utilization. >> >> === Inexperience with Open Source === >> The initial committers include PMC members and committers of other Apache >> projects. They have experience with open source projects, starting from >> their incubation to the top-level. They have been involved in the open >> source development process, and are familiar with releasing code under an >> open source license. >> >> === Homogeneous Developers === >> The initial set of committers is from a limited set of organizations, but we >> expect to attract new contributors from diverse organizations and will thus >> grow organically once approved for incubation. Our prior experience with >> other open source projects will help various contributors to actively >> participate in our project. >> >> === Reliance on Salaried Developers === >> Many developers are from Seoul National University. This is not applicable. >> >> === Relationships with Other Apache Products === >> Coral positions itself among multiple Apache products. It runs on Apache >> REEF for container management. It also utilizes many useful development >> tools including Apache Maven, Apache Log4J, and multiple Apache Commons >> components. Coral supports the Apache Beam programming model for user >> applications. We are currently working on supporting the Apache Spark >> programming APIs as well. >> >> === An Excessive Fascination with the Apache Brand === >> We hope to make Coral a powerful system for data processing, meeting various >> needs for different deployment characteristics, under a more variety of >> environments. We see the limitations of simply putting code on GitHub, and >> we believe the Apache community will help the growth of Coral for the >> project to become a positively impactful and innovative open source >> software. We believe Coral is a great fit for the Apache Software Foundation >> due to the collaboration it aims to achieve from the big data processing >> community. >> >> == Documentation == >> The current documentation for Coral is at https://snuspl.github.io/coral/. >> >> == Initial Source == >> The Coral codebase is currently hosted at https://github.com/snuspl/coral. >> >> == External Dependencies == >> To the best of our knowledge, all Coral dependencies are distributed under >> Apache compatible licenses. Upon acceptance to the incubator, we would begin >> a thorough analysis of all transitive dependencies to verify this fact and >> further introduce license checking into the build and release process. >> >> == Cryptography == >> Not applicable. >> >> == Required Resources == >> === Mailing Lists === >> We will operate two mailing lists as follows: >> * Coral PMC discussions: priv...@coral.incubator.apache.org >> * Coral developers: d...@coral.incubator.apache.org >> >> === Git Repositories === >> Upon incubation: https://github.com/apache/incubator-coral. >> After the incubation, we would like to move the existing repo >> https://github.com/snuspl/coral to the Apache infrastructure >> >> === Issue Tracking === >> Coral currently tracks its issues using the Github issue tracker: >> https://github.com/snuspl/coral/issues. We plan to migrate to Apache JIRA. >> >> == Initial Committers == >> * Byung-Gon Chun >> * Jeongyoon Eo >> * Geon-Woo Kim >> * Joo Yeon Kim >> * Gyewon Lee >> * Jung-Gil Lee >> * Sanha Lee >> * Wooyeon Lee >> * Yunseong Lee >> * JangHo Seo >> * Won Wook Song >> * Taegeon Um >> * Youngseok Yang >> >> == Affiliations == >> * SNU (Seoul National University) >> * Byung-Gon Chun >> * Jeongyoon Eo >> * Geon-Woo Kim >> * Gyewon Lee >> * Sanha Lee >> * Wooyeon Lee >> * Yunseong Lee >> * JangHo Seo >> * Won Wook Song >> * Taegeon Um >> * Youngseok Yang >> >> * LG >> * Jung-Gil Lee >> >> * Samsung >> * Joo Yeon Kim >> >> * Viva Republica >> * Geon-Woo Kim >> >> == Sponsors == >> === Champions === >> Byung-Gon Chun >> >> === Mentors === >> * Hyunsik Choi >> * Byung-Gon Chun >> * Jean-Baptiste Onofré >> * Markus Weimer >> * Reynold Xin >> >> === Sponsoring Entity === >> The Apache Incubator >> >> >> Thanks! >> Byung-Gon Chun >> > > > > -- > Byung-Gon Chun Craig L Russell Secretary, Apache Software Foundation c...@apache.org http://db.apache.org/jdo --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
