+1 (non-binding) Regards, Hao
On Fri, Jan 29, 2016 at 11:32 AM, Johan Edstrom <seij...@gmail.com> wrote: > +1 > > > On Jan 28, 2016, at 6:34 PM, Naresh Agarwal <naresh.agar...@gmail.com> > wrote: > > > > +1 (non-binding) > > > > Thanks > > Naresh > > On 29 Jan 2016 06:18, "Hadrian Zbarcea" <hzbar...@gmail.com> wrote: > > > >> +1 (binding) > >> > >> Man, congrats on a job fantastically well done. This is ASF incubator > >> participation at its best. > >> > >> Expectations are high now. I am looking forward to exemplary governance > >> and speedy graduation. > >> > >> Best of luck, > >> Hadrian > >> > >> On 01/28/2016 09:28 AM, Jean-Baptiste Onofré wrote: > >> > >>> Hi, > >>> > >>> the Beam proposal (initially Dataflow) was proposed last week. > >>> > >>> The complete discussion thread is available here: > >>> > >>> > >>> > http://mail-archives.apache.org/mod_mbox/incubator-general/201601.mbox/%3CCA%2B%3DKJmvj4wyosNTXVpnsH8PhS7jEyzkZngc682rGgZ3p28L42Q%40mail.gmail.com%3E > >>> > >>> > >>> As reminder the BeamProposal is here: > >>> > >>> https://wiki.apache.org/incubator/BeamProposal > >>> > >>> Regarding all the great feedbacks we received on the mailing list, we > >>> think it's time to call a vote to accept Beam into the Incubator. > >>> > >>> Please cast your vote to: > >>> [] +1 - accept Apache Beam as a new incubating project > >>> [] 0 - not sure > >>> [] -1 - do not accept the Apache Beam project (because: ...) > >>> > >>> Thanks, > >>> Regards > >>> JB > >>> ---- > >>> ## page was renamed from DataflowProposal > >>> = Apache Beam = > >>> > >>> == Abstract == > >>> > >>> Apache Beam is an open source, unified model and set of > >>> language-specific SDKs for defining and executing data processing > >>> workflows, and also data ingestion and integration flows, supporting > >>> Enterprise Integration Patterns (EIPs) and Domain Specific Languages > >>> (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch > >>> and streaming data processing and can run on a number of runtimes like > >>> Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud > service). > >>> Beam also brings DSL in different languages, allowing users to easily > >>> implement their data integration processes. > >>> > >>> == Proposal == > >>> > >>> Beam is a simple, flexible, and powerful system for distributed data > >>> processing at any scale. Beam provides a unified programming model, a > >>> software development kit to define and construct data processing > >>> pipelines, and runners to execute Beam pipelines in several runtime > >>> engines, like Apache Spark, Apache Flink, or Google Cloud Dataflow. > Beam > >>> can be used for a variety of streaming or batch data processing goals > >>> including ETL, stream analysis, and aggregate computation. The > >>> underlying programming model for Beam provides MapReduce-like > >>> parallelism, combined with support for powerful data windowing, and > >>> fine-grained correctness control. > >>> > >>> == Background == > >>> > >>> Beam started as a set of Google projects (Google Cloud Dataflow) > focused > >>> on making data processing easier, faster, and less costly. The Beam > >>> model is a successor to MapReduce, FlumeJava, and Millwheel inside > >>> Google and is focused on providing a unified solution for batch and > >>> stream processing. These projects on which Beam is based have been > >>> published in several papers made available to the public: > >>> > >>> * MapReduce - http://research.google.com/archive/mapreduce.html > >>> * Dataflow model - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf > >>> * FlumeJava - http://research.google.com/pubs/pub35650.html > >>> * MillWheel - http://research.google.com/pubs/pub41378.html > >>> > >>> Beam was designed from the start to provide a portable programming > >>> layer. When you define a data processing pipeline with the Beam model, > >>> you are creating a job which is capable of being processed by any > number > >>> of Beam processing engines. Several engines have been developed to run > >>> Beam pipelines in other open source runtimes, including a Beam runner > >>> for Apache Flink and Apache Spark. There is also a “direct runner”, for > >>> execution on the developer machine (mainly for dev/debug purposes). > >>> Another runner allows a Beam program to run on a managed service, > Google > >>> Cloud Dataflow, in Google Cloud Platform. The Dataflow Java SDK is > >>> already available on GitHub, and independent from the Google Cloud > >>> Dataflow service. Another Python SDK is currently in active > development. > >>> > >>> In this proposal, the Beam SDKs, model, and a set of runners will be > >>> submitted as an OSS project under the ASF. The runners which are a part > >>> of this proposal include those for Spark (from Cloudera), Flink (from > >>> data Artisans), and local development (from Google); the Google Cloud > >>> Dataflow service runner is not included in this proposal. Further > >>> references to Beam will refer to the Dataflow model, SDKs, and runners > >>> which are a part of this proposal (Apache Beam) only. The initial > >>> submission will contain the already-released Java SDK; Google intends > to > >>> submit the Python SDK later in the incubation process. The Google Cloud > >>> Dataflow service will continue to be one of many runners for Beam, > built > >>> on Google Cloud Platform, to run Beam pipelines. Necessarily, Cloud > >>> Dataflow will develop against the Apache project additions, updates, > and > >>> changes. Google Cloud Dataflow will become one user of Apache Beam and > >>> will participate in the project openly and publicly. > >>> > >>> The Beam programming model has been designed with simplicity, > >>> scalability, and speed as key tenants. In the Beam model, you only need > >>> to think about four top-level concepts when constructing your data > >>> processing job: > >>> > >>> * Pipelines - The data processing job made of a series of computations > >>> including input, processing, and output > >>> * PCollections - Bounded (or unbounded) datasets which represent the > >>> input, intermediate and output data in pipelines > >>> * PTransforms - A data processing step in a pipeline in which one or > >>> more PCollections are an input and output > >>> * I/O Sources and Sinks - APIs for reading and writing data which are > >>> the roots and endpoints of the pipeline > >>> > >>> == Rationale == > >>> > >>> With Google Dataflow, Google intended to develop a framework which > >>> allowed developers to be maximally productive in defining the > >>> processing, and then be able to execute the program at various levels > of > >>> latency/cost/completeness without re-architecting or re-writing it. > This > >>> goal was informed by Google’s past experience developing several > >>> models, frameworks, and tools useful for large-scale and distributed > >>> data processing. While Google has previously published papers > describing > >>> some of its technologies, Google decided to take a different approach > >>> with Dataflow. Google open-sourced the SDK and model alongside > >>> commercialization of the idea and ahead of publishing papers on the > >>> topic. As a result, a number of open source runtimes exist for > Dataflow, > >>> such as the Apache Flink and Apache Spark runners. > >>> > >>> We believe that submitting Beam as an Apache project will provide an > >>> immediate, worthwhile, and substantial contribution to the open source > >>> community. As an incubating project, we believe Dataflow will have a > >>> better opportunity to provide a meaningful contribution to OSS and also > >>> integrate with other Apache projects. > >>> > >>> In the long term, we believe Beam can be a powerful abstraction layer > >>> for data processing. By providing an abstraction layer for data > >>> pipelines and processing, data workflows can be increasingly portable, > >>> resilient to breaking changes in tooling, and compatible across many > >>> execution engines, runtimes, and open source projects. > >>> > >>> == Initial Goals == > >>> > >>> We are breaking our initial goals into immediate (< 2 months), > >>> short-term (2-4 months), and intermediate-term (> 4 months). > >>> > >>> Our immediate goals include the following: > >>> > >>> * Plan for reconciling the Dataflow Java SDK and various runners into > >>> one project > >>> * Plan for refactoring the existing Java SDK for better extensibility > >>> by SDK and runner writers > >>> * Validating all dependencies are ASL 2.0 or compatible > >>> * Understanding and adapting to the Apache development process > >>> > >>> Our short-term goals include: > >>> > >>> * Moving the newly-merged lists, and build utilities to Apache > >>> * Start refactoring codebase and move code to Apache Git repo > >>> * Continue development of new features, functions, and fixes in the > >>> Dataflow Java SDK, and Dataflow runners > >>> * Cleaning up the Dataflow SDK sources and crafting a roadmap and plan > >>> for how to include new major ideas, modules, and runtimes > >>> * Establishment of easy and clear build/test framework for Dataflow > >>> and associated runtimes; creation of testing, rollback, and validation > >>> policy > >>> * Analysis and design for work needed to make Beam a better data > >>> processing abstraction layer for multiple open source frameworks and > >>> environments > >>> > >>> Finally, we have a number of intermediate-term goals: > >>> > >>> * Roadmapping, planning, and execution of integrations with other OSS > >>> and non-OSS projects/products > >>> * Inclusion of additional SDK for Python, which is under active > >>> development > >>> > >>> == Current Status == > >>> > >>> === Meritocracy === > >>> > >>> Dataflow was initially developed based on ideas from many employees > >>> within Google. As an ASL OSS project on GitHub, the Dataflow SDK has > >>> received contributions from data Artisans, Cloudera Labs, and other > >>> individual developers. As a project under incubation, we are committed > >>> to expanding our effort to build an environment which supports a > >>> meritocracy. We are focused on engaging the community and other related > >>> projects for support and contributions. Moreover, we are committed to > >>> ensure contributors and committers to Dataflow come from a broad mix of > >>> organizations through a merit-based decision process during incubation. > >>> We believe strongly in the Beam model and are committed to growing an > >>> inclusive community of Beam contributors. > >>> > >>> === Community === > >>> > >>> The core of the Dataflow Java SDK has been developed by Google for use > >>> with Google Cloud Dataflow. Google has active community engagement in > >>> the SDK GitHub repository > >>> (https://github.com/GoogleCloudPlatform/DataflowJavaSDK), on Stack > >>> Overflow > >>> (http://stackoverflow.com/questions/tagged/google-cloud-dataflow) and > >>> has had contributions from a number of organizations and indivuduals. > >>> > >>> Everyday, Cloud Dataflow is actively used by a number of organizations > >>> and institutions for batch and stream processing of data. We believe > >>> acceptance will allow us to consolidate existing Dataflow-related work, > >>> grow the Dataflow community, and deepen connections between Dataflow > and > >>> other open source projects. > >>> > >>> === Core Developers === > >>> > >>> The core developers for Dataflow and the Dataflow runners are: > >>> > >>> * Frances Perry > >>> * Tyler Akidau > >>> * Davor Bonaci > >>> * Luke Cwik > >>> * Ben Chambers > >>> * Kenn Knowles > >>> * Dan Halperin > >>> * Daniel Mills > >>> * Mark Shields > >>> * Craig Chambers > >>> * Maximilian Michels > >>> * Tom White > >>> * Josh Wills > >>> * Robert Bradshaw > >>> > >>> === Alignment === > >>> > >>> The Beam SDK can be used to create Beam pipelines which can be executed > >>> on Apache Spark or Apache Flink. Beam is also related to other Apache > >>> projects, such as Apache Crunch. We plan on expanding functionality for > >>> Beam runners, support for additional domain specific languages, and > >>> increased portability so Beam is a powerful abstraction layer for data > >>> processing. > >>> > >>> == Known Risks == > >>> > >>> === Orphaned Products === > >>> > >>> The Dataflow SDK is presently used by several organizations, from small > >>> startups to Fortune 100 companies, to construct production pipelines > >>> which are executed in Google Cloud Dataflow. Google has a long-term > >>> commitment to advance the Dataflow SDK; moreover, Dataflow is seeing > >>> increasing interest, development, and adoption from organizations > >>> outside of Google. > >>> > >>> === Inexperience with Open Source === > >>> > >>> Google believes strongly in open source and the exchange of information > >>> to advance new ideas and work. Examples of this commitment are active > >>> OSS projects such as Chromium (https://www.chromium.org) and > Kubernetes > >>> (http://kubernetes.io/). With Dataflow, we have tried to be > increasingly > >>> open and forward-looking; we have published a paper in the VLDB > >>> conference describing the Dataflow model > >>> (http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to > >>> release the Dataflow SDK as open source software with the launch of > >>> Cloud Dataflow. Our submission to the Apache Software Foundation is a > >>> logical extension of our commitment to open source software. > >>> > >>> === Homogeneous Developers === > >>> > >>> The majority of committers in this proposal belong to Google due to the > >>> fact that Dataflow has emerged from several internal Google projects. > >>> This proposal also includes committers outside of Google who are > >>> actively involved with other Apache projects, such as Hadoop, Flink, > and > >>> Spark. We expect our entry into incubation will allow us to expand the > >>> number of individuals and organizations participating in Dataflow > >>> development. Additionally, separation of the Dataflow SDK from Google > >>> Cloud Dataflow allows us to focus on the open source SDK and model and > >>> do what is best for this project. > >>> > >>> === Reliance on Salaried Developers === > >>> > >>> The Dataflow SDK and Dataflow runners have been developed primarily by > >>> salaried developers supporting the Google Cloud Dataflow project. While > >>> the Dataflow SDK and Cloud Dataflow have been developed by different > >>> teams (and this proposal would reinforce that separation) we expect our > >>> initial set of developers will still primarily be salaried. > Contribution > >>> has not been exclusively from salaried developers, however. For > example, > >>> the contrib directory of the Dataflow SDK > >>> ( > >>> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib > >>> ) > >>> contains items from free-time contributors. Moreover, seperate > projects, > >>> such as ScalaFlow (https://github.com/darkjh/scalaflow) have been > >>> created around the Dataflow model and SDK. We expect our reliance on > >>> salaried developers will decrease over time during incubation. > >>> > >>> === Relationship with other Apache products === > >>> > >>> Dataflow directly interoperates with or utilizes several existing > Apache > >>> projects. > >>> > >>> * Build > >>> * Apache Maven > >>> * Data I/O, Libraries > >>> * Apache Avro > >>> * Apache Commons > >>> * Dataflow runners > >>> * Apache Flink > >>> * Apache Spark > >>> > >>> Beam when used in batch mode shares similarities with Apache Crunch; > >>> however, Beam is focused on a model, SDK, and abstraction layer beyond > >>> Spark and Hadoop (MapReduce.) One key goal of Beam is to provide an > >>> intermediate abstraction layer which can easily be implemented and > >>> utilized across several different processing frameworks. > >>> > >>> === An excessive fascination with the Apache brand === > >>> > >>> With this proposal we are not seeking attention or publicity. Rather, > we > >>> firmly believe in the Beam model, SDK, and the ability to make Beam a > >>> powerful yet simple framework for data processing. While the Dataflow > >>> SDK and model have been open source, we believe putting code on GitHub > >>> can only go so far. We see the Apache community, processes, and mission > >>> as critical for ensuring the Beam SDK and model are truly > >>> community-driven, positively impactful, and innovative open source > >>> software. While Google has taken a number of steps to advance its > >>> various open source projects, we believe Beam is a great fit for the > >>> Apache Software Foundation due to its focus on data processing and its > >>> relationships to existing ASF projects. > >>> > >>> == Documentation == > >>> > >>> The following documentation is relevant to this proposal. Relevant > >>> portion of the documentation will be contributed to the Apache Beam > >>> project. > >>> > >>> * Dataflow website: https://cloud.google.com/dataflow > >>> * Dataflow programming model: > >>> https://cloud.google.com/dataflow/model/programming-model > >>> * Codebases > >>> * Dataflow Java SDK: > >>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK > >>> * Flink Dataflow runner: > >>> https://github.com/dataArtisans/flink-dataflow > >>> * Spark Dataflow runner: https://github.com/cloudera/spark-dataflow > >>> * Dataflow Java SDK issue tracker: > >>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues > >>> * google-cloud-dataflow tag on Stack Overflow: > >>> http://stackoverflow.com/questions/tagged/google-cloud-dataflow > >>> > >>> == Initial Source == > >>> > >>> The initial source for Beam which we will submit to the Apache > >>> Foundation will include several related projects which are currently > >>> hosted on the GitHub repositories: > >>> > >>> * Dataflow Java SDK > >>> (https://github.com/GoogleCloudPlatform/DataflowJavaSDK) > >>> * Flink Dataflow runner ( > https://github.com/dataArtisans/flink-dataflow > >>> ) > >>> * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow) > >>> > >>> These projects have always been Apache 2.0 licensed. We intend to > bundle > >>> all of these repositories since they are all complimentary and should > be > >>> maintained in one project. Prior to our submission, we will combine all > >>> of these projects into a new git repository. > >>> > >>> == Source and Intellectual Property Submission Plan == > >>> > >>> The source for the Dataflow SDK and the three runners (Spark, Flink, > >>> Google Cloud Dataflow) are already licensed under an Apache 2 license. > >>> > >>> * Dataflow SDK - > >>> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE > >>> * Flink runner - > >>> https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE > >>> * Spark runner - > >>> https://github.com/cloudera/spark-dataflow/blob/master/LICENSE > >>> > >>> Contributors to the Dataflow SDK have also signed the Google Individual > >>> Contributor License Agreement > >>> (https://cla.developers.google.com/about/google-individual) in order > to > >>> contribute to the project. > >>> > >>> With respect to trademark rights, Google does not hold a trademark on > >>> the phrase “Dataflow.” Based on feedback and guidance we receive during > >>> the incubation process, we are open to renaming the project if > necessary > >>> for trademark or other concerns. > >>> > >>> == External Dependencies == > >>> > >>> All external dependencies are licensed under an Apache 2.0 or > >>> Apache-compatible license. As we grow the Beam community we will > >>> configure our build process to require and validate all contributions > >>> and dependencies are licensed under the Apache 2.0 license or are under > >>> an Apache-compatible license. > >>> > >>> == Required Resources == > >>> > >>> === Mailing Lists === > >>> > >>> We currently use a mix of mailing lists. We will migrate our existing > >>> mailing lists to the following: > >>> > >>> * d...@beam.incubator.apache.org > >>> * u...@beam.incubator.apache.org > >>> * priv...@beam.incubator.apache.org > >>> * comm...@beam.incubator.apache.org > >>> > >>> === Source Control === > >>> > >>> The Dataflow team currently uses Git and would like to continue to do > >>> so. We request a Git repository for Beam with mirroring to GitHub > enabled. > >>> > >>> * https://git-wip-us.apache.org/repos/asf/incubator-beam.git > >>> > >>> === Issue Tracking === > >>> > >>> We request the creation of an Apache-hosted JIRA. The Dataflow project > >>> is currently using both a public GitHub issue tracker and internal > >>> Google issue tracking. We will migrate and combine from these two > >>> sources to the Apache JIRA. > >>> > >>> * Jira ID: BEAM > >>> > >>> == Initial Committers == > >>> > >>> * Aljoscha Krettek [aljos...@apache.org] > >>> * Amit Sela [amitsel...@gmail.com] > >>> * Ben Chambers [bchamb...@google.com] > >>> * Craig Chambers [chamb...@google.com] > >>> * Dan Halperin [dhalp...@google.com] > >>> * Davor Bonaci [da...@google.com] > >>> * Frances Perry [f...@google.com] > >>> * James Malone [jamesmal...@google.com] > >>> * Jean-Baptiste Onofré [jbono...@apache.org] > >>> * Josh Wills [jwi...@apache.org] > >>> * Kostas Tzoumas [kos...@data-artisans.com] > >>> * Kenneth Knowles [k...@google.com] > >>> * Luke Cwik [lc...@google.com] > >>> * Maximilian Michels [m...@apache.org] > >>> * Stephan Ewen [step...@data-artisans.com] > >>> * Tom White [t...@cloudera.com] > >>> * Tyler Akidau [taki...@google.com] > >>> * Robert Bradshaw [rober...@google.com] > >>> > >>> == Additional Interested Contributors == > >>> > >>> * Debo Dutta [dedu...@cisco.com] > >>> * Henry Saputra [hsapu...@apache.org] > >>> * Taylor Goetz [ptgo...@gmail.com] > >>> * James Carman [ja...@carmanconsulting.com] > >>> * Joe Witt [joew...@apache.org] > >>> * Vaibhav Gumashta [vgumas...@hortonworks.com] > >>> * Prasanth Jayachandran [pjayachand...@hortonworks.com] > >>> * Johan Edstrom [seij...@gmail.com] > >>> * Hugo Louro [hmclo...@gmail.com] > >>> * Krzysztof Sobkowiak [krzys.sobkow...@gmail.com] > >>> * Jeff Genender [jgenen...@apache.org] > >>> * Edward J. Yoon [edward.y...@samsung.com] > >>> * Hao Chen [h...@apache.org] > >>> * Byung-Gon Chun [bgc...@gmail.com] > >>> * Charitha Elvitigala [charit...@apache.org] > >>> * Alexander Bezzubov [b...@apache.org] > >>> * Tsuyoshi Ozawa [oz...@apache.org] > >>> * Mayank Bansal [maban...@gmail.com] > >>> * Supun Kamburugamuve [su...@apache.org] > >>> * Matthias Wessendorf [mat...@apache.org] > >>> * Felix Cheung [felixche...@apache.org] > >>> * Ajay Yadava [ajay.ya...@inmobi.com] > >>> * Liang Chen [chenliang...@huawei.com] > >>> * Renaud Richardet [renaud (at) apache (dot) org] > >>> * Bakey Pan [bakey1...@gmail.com] > >>> * Andreas Neumann [a...@apache.org] > >>> * Suresh Marru [sma...@apache.org] > >>> * Hadrian Zbarcea [hzbar...@gmail.com] > >>> > >>> == Affiliations == > >>> > >>> The initial committers are from six organizations. Google developed > >>> Dataflow and the Dataflow SDK, data Artisans developed the Flink > runner, > >>> and Cloudera (Labs) developed the Spark runner. > >>> > >>> * Cloudera > >>> * Tom White > >>> * Data Artisans > >>> * Aljoscha Krettek > >>> * Kostas Tzoumas > >>> * Maximilian Michels > >>> * Stephan Ewen > >>> * Google > >>> * Ben Chambers > >>> * Dan Halperin > >>> * Davor Bonaci > >>> * Frances Perry > >>> * James Malone > >>> * Kenneth Knowles > >>> * Luke Cwik > >>> * Tyler Akidau > >>> * Robert Bradshaw > >>> * PayPal > >>> * Amit Sela > >>> * Slack > >>> * Josh Wills > >>> * Talend > >>> * Jean-Baptiste Onofré > >>> > >>> == Sponsors == > >>> > >>> === Champion === > >>> > >>> * Jean-Baptiste Onofre [jbono...@apache.org] > >>> > >>> === Nominated Mentors === > >>> > >>> * Jean-Baptiste Onofre [jbono...@apache.org] > >>> * Jim Jagielski [j...@apache.org] > >>> * Venkatesh Seetharam [venkat...@apache.org] > >>> * Bertrand Delacretaz [bdelacre...@apache.org] > >>> * Ted Dunning [tdunn...@apache.org] > >>> > >>> === Sponsoring Entity === > >>> > >>> The Apache Incubator > >>> ---- > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >>> For additional commands, e-mail: general-h...@incubator.apache.org > >>> > >>> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> For additional commands, e-mail: general-h...@incubator.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
