+1 (non-binding) 2016-01-29 23:51 GMT+08:00 Luke Han <luke...@apache.org>:
> +1 (non-binding) > > On Fri, Jan 29, 2016 at 6:31 PM, Tom Barber <tom.bar...@meteorite.bi> > wrote: > > > +1 (binding) > > > > On Fri, Jan 29, 2016 at 10:03 AM, Tom White <tom.e.wh...@gmail.com> > wrote: > > > > > +1 (binding) > > > > > > Tom > > > > > > On Thu, Jan 28, 2016 at 2:28 PM, Jean-Baptiste Onofré <j...@nanthrax.net > > > > > wrote: > > > > Hi, > > > > > > > > the Beam proposal (initially Dataflow) was proposed last week. > > > > > > > > The complete discussion thread is available here: > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/incubator-general/201601.mbox/%3CCA%2B%3DKJmvj4wyosNTXVpnsH8PhS7jEyzkZngc682rGgZ3p28L42Q%40mail.gmail.com%3E > > > > > > > > As reminder the BeamProposal is here: > > > > > > > > https://wiki.apache.org/incubator/BeamProposal > > > > > > > > Regarding all the great feedbacks we received on the mailing list, we > > > think > > > > it's time to call a vote to accept Beam into the Incubator. > > > > > > > > Please cast your vote to: > > > > [] +1 - accept Apache Beam as a new incubating project > > > > [] 0 - not sure > > > > [] -1 - do not accept the Apache Beam project (because: ...) > > > > > > > > Thanks, > > > > Regards > > > > JB > > > > ---- > > > > ## page was renamed from DataflowProposal > > > > = Apache Beam = > > > > > > > > == Abstract == > > > > > > > > Apache Beam is an open source, unified model and set of > > language-specific > > > > SDKs for defining and executing data processing workflows, and also > > data > > > > ingestion and integration flows, supporting Enterprise Integration > > > Patterns > > > > (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines > > simplify > > > the > > > > mechanics of large-scale batch and streaming data processing and can > > run > > > on > > > > a number of runtimes like Apache Flink, Apache Spark, and Google > Cloud > > > > Dataflow (a cloud service). Beam also brings DSL in different > > languages, > > > > allowing users to easily implement their data integration processes. > > > > > > > > == Proposal == > > > > > > > > Beam is a simple, flexible, and powerful system for distributed data > > > > processing at any scale. Beam provides a unified programming model, a > > > > software development kit to define and construct data processing > > > pipelines, > > > > and runners to execute Beam pipelines in several runtime engines, > like > > > > Apache Spark, Apache Flink, or Google Cloud Dataflow. Beam can be > used > > > for a > > > > variety of streaming or batch data processing goals including ETL, > > stream > > > > analysis, and aggregate computation. The underlying programming model > > for > > > > Beam provides MapReduce-like parallelism, combined with support for > > > powerful > > > > data windowing, and fine-grained correctness control. > > > > > > > > == Background == > > > > > > > > Beam started as a set of Google projects (Google Cloud Dataflow) > > focused > > > on > > > > making data processing easier, faster, and less costly. The Beam > model > > > is a > > > > successor to MapReduce, FlumeJava, and Millwheel inside Google and is > > > > focused on providing a unified solution for batch and stream > > processing. > > > > These projects on which Beam is based have been published in several > > > papers > > > > made available to the public: > > > > > > > > * MapReduce - http://research.google.com/archive/mapreduce.html > > > > * Dataflow model - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf > > > > * FlumeJava - http://research.google.com/pubs/pub35650.html > > > > * MillWheel - http://research.google.com/pubs/pub41378.html > > > > > > > > Beam was designed from the start to provide a portable programming > > layer. > > > > When you define a data processing pipeline with the Beam model, you > are > > > > creating a job which is capable of being processed by any number of > > Beam > > > > processing engines. Several engines have been developed to run Beam > > > > pipelines in other open source runtimes, including a Beam runner for > > > Apache > > > > Flink and Apache Spark. There is also a “direct runner”, for > execution > > on > > > > the developer machine (mainly for dev/debug purposes). Another runner > > > allows > > > > a Beam program to run on a managed service, Google Cloud Dataflow, in > > > Google > > > > Cloud Platform. The Dataflow Java SDK is already available on GitHub, > > and > > > > independent from the Google Cloud Dataflow service. Another Python > SDK > > is > > > > currently in active development. > > > > > > > > In this proposal, the Beam SDKs, model, and a set of runners will be > > > > submitted as an OSS project under the ASF. The runners which are a > part > > > of > > > > this proposal include those for Spark (from Cloudera), Flink (from > data > > > > Artisans), and local development (from Google); the Google Cloud > > Dataflow > > > > service runner is not included in this proposal. Further references > to > > > Beam > > > > will refer to the Dataflow model, SDKs, and runners which are a part > of > > > this > > > > proposal (Apache Beam) only. The initial submission will contain the > > > > already-released Java SDK; Google intends to submit the Python SDK > > later > > > in > > > > the incubation process. The Google Cloud Dataflow service will > continue > > > to > > > > be one of many runners for Beam, built on Google Cloud Platform, to > run > > > Beam > > > > pipelines. Necessarily, Cloud Dataflow will develop against the > Apache > > > > project additions, updates, and changes. Google Cloud Dataflow will > > > become > > > > one user of Apache Beam and will participate in the project openly > and > > > > publicly. > > > > > > > > The Beam programming model has been designed with simplicity, > > > scalability, > > > > and speed as key tenants. In the Beam model, you only need to think > > about > > > > four top-level concepts when constructing your data processing job: > > > > > > > > * Pipelines - The data processing job made of a series of > computations > > > > including input, processing, and output > > > > * PCollections - Bounded (or unbounded) datasets which represent the > > > input, > > > > intermediate and output data in pipelines > > > > * PTransforms - A data processing step in a pipeline in which one or > > > more > > > > PCollections are an input and output > > > > * I/O Sources and Sinks - APIs for reading and writing data which > are > > > the > > > > roots and endpoints of the pipeline > > > > > > > > == Rationale == > > > > > > > > With Google Dataflow, Google intended to develop a framework which > > > allowed > > > > developers to be maximally productive in defining the processing, and > > > then > > > > be able to execute the program at various levels of > > > > latency/cost/completeness without re-architecting or re-writing it. > > This > > > > goal was informed by Google’s past experience developing several > > models, > > > > frameworks, and tools useful for large-scale and distributed data > > > > processing. While Google has previously published papers describing > > some > > > of > > > > its technologies, Google decided to take a different approach with > > > Dataflow. > > > > Google open-sourced the SDK and model alongside commercialization of > > the > > > > idea and ahead of publishing papers on the topic. As a result, a > number > > > of > > > > open source runtimes exist for Dataflow, such as the Apache Flink and > > > Apache > > > > Spark runners. > > > > > > > > We believe that submitting Beam as an Apache project will provide an > > > > immediate, worthwhile, and substantial contribution to the open > source > > > > community. As an incubating project, we believe Dataflow will have a > > > better > > > > opportunity to provide a meaningful contribution to OSS and also > > > integrate > > > > with other Apache projects. > > > > > > > > In the long term, we believe Beam can be a powerful abstraction layer > > for > > > > data processing. By providing an abstraction layer for data pipelines > > and > > > > processing, data workflows can be increasingly portable, resilient to > > > > breaking changes in tooling, and compatible across many execution > > > engines, > > > > runtimes, and open source projects. > > > > > > > > == Initial Goals == > > > > > > > > We are breaking our initial goals into immediate (< 2 months), > > short-term > > > > (2-4 months), and intermediate-term (> 4 months). > > > > > > > > Our immediate goals include the following: > > > > > > > > * Plan for reconciling the Dataflow Java SDK and various runners > into > > > one > > > > project > > > > * Plan for refactoring the existing Java SDK for better > extensibility > > by > > > > SDK and runner writers > > > > * Validating all dependencies are ASL 2.0 or compatible > > > > * Understanding and adapting to the Apache development process > > > > > > > > Our short-term goals include: > > > > > > > > * Moving the newly-merged lists, and build utilities to Apache > > > > * Start refactoring codebase and move code to Apache Git repo > > > > * Continue development of new features, functions, and fixes in the > > > > Dataflow Java SDK, and Dataflow runners > > > > * Cleaning up the Dataflow SDK sources and crafting a roadmap and > plan > > > for > > > > how to include new major ideas, modules, and runtimes > > > > * Establishment of easy and clear build/test framework for Dataflow > > and > > > > associated runtimes; creation of testing, rollback, and validation > > policy > > > > * Analysis and design for work needed to make Beam a better data > > > processing > > > > abstraction layer for multiple open source frameworks and > environments > > > > > > > > Finally, we have a number of intermediate-term goals: > > > > > > > > * Roadmapping, planning, and execution of integrations with other > OSS > > > and > > > > non-OSS projects/products > > > > * Inclusion of additional SDK for Python, which is under active > > > development > > > > > > > > == Current Status == > > > > > > > > === Meritocracy === > > > > > > > > Dataflow was initially developed based on ideas from many employees > > > within > > > > Google. As an ASL OSS project on GitHub, the Dataflow SDK has > received > > > > contributions from data Artisans, Cloudera Labs, and other individual > > > > developers. As a project under incubation, we are committed to > > expanding > > > our > > > > effort to build an environment which supports a meritocracy. We are > > > focused > > > > on engaging the community and other related projects for support and > > > > contributions. Moreover, we are committed to ensure contributors and > > > > committers to Dataflow come from a broad mix of organizations > through a > > > > merit-based decision process during incubation. We believe strongly > in > > > the > > > > Beam model and are committed to growing an inclusive community of > Beam > > > > contributors. > > > > > > > > === Community === > > > > > > > > The core of the Dataflow Java SDK has been developed by Google for > use > > > with > > > > Google Cloud Dataflow. Google has active community engagement in the > > SDK > > > > GitHub repository ( > > > https://github.com/GoogleCloudPlatform/DataflowJavaSDK), > > > > on Stack Overflow > > > > (http://stackoverflow.com/questions/tagged/google-cloud-dataflow) > and > > > has > > > > had contributions from a number of organizations and indivuduals. > > > > > > > > Everyday, Cloud Dataflow is actively used by a number of > organizations > > > and > > > > institutions for batch and stream processing of data. We believe > > > acceptance > > > > will allow us to consolidate existing Dataflow-related work, grow the > > > > Dataflow community, and deepen connections between Dataflow and other > > > open > > > > source projects. > > > > > > > > === Core Developers === > > > > > > > > The core developers for Dataflow and the Dataflow runners are: > > > > > > > > * Frances Perry > > > > * Tyler Akidau > > > > * Davor Bonaci > > > > * Luke Cwik > > > > * Ben Chambers > > > > * Kenn Knowles > > > > * Dan Halperin > > > > * Daniel Mills > > > > * Mark Shields > > > > * Craig Chambers > > > > * Maximilian Michels > > > > * Tom White > > > > * Josh Wills > > > > * Robert Bradshaw > > > > > > > > === Alignment === > > > > > > > > The Beam SDK can be used to create Beam pipelines which can be > executed > > > on > > > > Apache Spark or Apache Flink. Beam is also related to other Apache > > > projects, > > > > such as Apache Crunch. We plan on expanding functionality for Beam > > > runners, > > > > support for additional domain specific languages, and increased > > > portability > > > > so Beam is a powerful abstraction layer for data processing. > > > > > > > > == Known Risks == > > > > > > > > === Orphaned Products === > > > > > > > > The Dataflow SDK is presently used by several organizations, from > small > > > > startups to Fortune 100 companies, to construct production pipelines > > > which > > > > are executed in Google Cloud Dataflow. Google has a long-term > > commitment > > > to > > > > advance the Dataflow SDK; moreover, Dataflow is seeing increasing > > > interest, > > > > development, and adoption from organizations outside of Google. > > > > > > > > === Inexperience with Open Source === > > > > > > > > Google believes strongly in open source and the exchange of > information > > > to > > > > advance new ideas and work. Examples of this commitment are active > OSS > > > > projects such as Chromium (https://www.chromium.org) and Kubernetes > > > > (http://kubernetes.io/). With Dataflow, we have tried to be > > increasingly > > > > open and forward-looking; we have published a paper in the VLDB > > > conference > > > > describing the Dataflow model > > > > (http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to > > > release > > > > the Dataflow SDK as open source software with the launch of Cloud > > > Dataflow. > > > > Our submission to the Apache Software Foundation is a logical > extension > > > of > > > > our commitment to open source software. > > > > > > > > === Homogeneous Developers === > > > > > > > > The majority of committers in this proposal belong to Google due to > the > > > fact > > > > that Dataflow has emerged from several internal Google projects. This > > > > proposal also includes committers outside of Google who are actively > > > > involved with other Apache projects, such as Hadoop, Flink, and > Spark. > > > We > > > > expect our entry into incubation will allow us to expand the number > of > > > > individuals and organizations participating in Dataflow development. > > > > Additionally, separation of the Dataflow SDK from Google Cloud > Dataflow > > > > allows us to focus on the open source SDK and model and do what is > best > > > for > > > > this project. > > > > > > > > === Reliance on Salaried Developers === > > > > > > > > The Dataflow SDK and Dataflow runners have been developed primarily > by > > > > salaried developers supporting the Google Cloud Dataflow project. > While > > > the > > > > Dataflow SDK and Cloud Dataflow have been developed by different > teams > > > (and > > > > this proposal would reinforce that separation) we expect our initial > > set > > > of > > > > developers will still primarily be salaried. Contribution has not > been > > > > exclusively from salaried developers, however. For example, the > contrib > > > > directory of the Dataflow SDK > > > > ( > > > > > > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib > > > ) > > > > contains items from free-time contributors. Moreover, seperate > > projects, > > > > such as ScalaFlow (https://github.com/darkjh/scalaflow) have been > > > created > > > > around the Dataflow model and SDK. We expect our reliance on salaried > > > > developers will decrease over time during incubation. > > > > > > > > === Relationship with other Apache products === > > > > > > > > Dataflow directly interoperates with or utilizes several existing > > Apache > > > > projects. > > > > > > > > * Build > > > > * Apache Maven > > > > * Data I/O, Libraries > > > > * Apache Avro > > > > * Apache Commons > > > > * Dataflow runners > > > > * Apache Flink > > > > * Apache Spark > > > > > > > > Beam when used in batch mode shares similarities with Apache Crunch; > > > > however, Beam is focused on a model, SDK, and abstraction layer > beyond > > > Spark > > > > and Hadoop (MapReduce.) One key goal of Beam is to provide an > > > intermediate > > > > abstraction layer which can easily be implemented and utilized across > > > > several different processing frameworks. > > > > > > > > === An excessive fascination with the Apache brand === > > > > > > > > With this proposal we are not seeking attention or publicity. Rather, > > we > > > > firmly believe in the Beam model, SDK, and the ability to make Beam a > > > > powerful yet simple framework for data processing. While the Dataflow > > SDK > > > > and model have been open source, we believe putting code on GitHub > can > > > only > > > > go so far. We see the Apache community, processes, and mission as > > > critical > > > > for ensuring the Beam SDK and model are truly community-driven, > > > positively > > > > impactful, and innovative open source software. While Google has > taken > > a > > > > number of steps to advance its various open source projects, we > believe > > > Beam > > > > is a great fit for the Apache Software Foundation due to its focus on > > > data > > > > processing and its relationships to existing ASF projects. > > > > > > > > == Documentation == > > > > > > > > The following documentation is relevant to this proposal. Relevant > > > portion > > > > of the documentation will be contributed to the Apache Beam project. > > > > > > > > * Dataflow website: https://cloud.google.com/dataflow > > > > * Dataflow programming model: > > > > https://cloud.google.com/dataflow/model/programming-model > > > > * Codebases > > > > * Dataflow Java SDK: > > > > https://github.com/GoogleCloudPlatform/DataflowJavaSDK > > > > * Flink Dataflow runner: > > > https://github.com/dataArtisans/flink-dataflow > > > > * Spark Dataflow runner: > https://github.com/cloudera/spark-dataflow > > > > * Dataflow Java SDK issue tracker: > > > > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues > > > > * google-cloud-dataflow tag on Stack Overflow: > > > > http://stackoverflow.com/questions/tagged/google-cloud-dataflow > > > > > > > > == Initial Source == > > > > > > > > The initial source for Beam which we will submit to the Apache > > Foundation > > > > will include several related projects which are currently hosted on > the > > > > GitHub repositories: > > > > > > > > * Dataflow Java SDK > > > > (https://github.com/GoogleCloudPlatform/DataflowJavaSDK) > > > > * Flink Dataflow runner ( > > https://github.com/dataArtisans/flink-dataflow > > > ) > > > > * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow > ) > > > > > > > > These projects have always been Apache 2.0 licensed. We intend to > > bundle > > > all > > > > of these repositories since they are all complimentary and should be > > > > maintained in one project. Prior to our submission, we will combine > all > > > of > > > > these projects into a new git repository. > > > > > > > > == Source and Intellectual Property Submission Plan == > > > > > > > > The source for the Dataflow SDK and the three runners (Spark, Flink, > > > Google > > > > Cloud Dataflow) are already licensed under an Apache 2 license. > > > > > > > > * Dataflow SDK - > > > > > > > > > > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE > > > > * Flink runner - > > > > https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE > > > > * Spark runner - > > > > https://github.com/cloudera/spark-dataflow/blob/master/LICENSE > > > > > > > > Contributors to the Dataflow SDK have also signed the Google > Individual > > > > Contributor License Agreement > > > > (https://cla.developers.google.com/about/google-individual) in order > > to > > > > contribute to the project. > > > > > > > > With respect to trademark rights, Google does not hold a trademark on > > the > > > > phrase “Dataflow.” Based on feedback and guidance we receive during > the > > > > incubation process, we are open to renaming the project if necessary > > for > > > > trademark or other concerns. > > > > > > > > == External Dependencies == > > > > > > > > All external dependencies are licensed under an Apache 2.0 or > > > > Apache-compatible license. As we grow the Beam community we will > > > configure > > > > our build process to require and validate all contributions and > > > dependencies > > > > are licensed under the Apache 2.0 license or are under an > > > Apache-compatible > > > > license. > > > > > > > > == Required Resources == > > > > > > > > === Mailing Lists === > > > > > > > > We currently use a mix of mailing lists. We will migrate our existing > > > > mailing lists to the following: > > > > > > > > * d...@beam.incubator.apache.org > > > > * u...@beam.incubator.apache.org > > > > * priv...@beam.incubator.apache.org > > > > * comm...@beam.incubator.apache.org > > > > > > > > === Source Control === > > > > > > > > The Dataflow team currently uses Git and would like to continue to do > > > so. We > > > > request a Git repository for Beam with mirroring to GitHub enabled. > > > > > > > > * https://git-wip-us.apache.org/repos/asf/incubator-beam.git > > > > > > > > === Issue Tracking === > > > > > > > > We request the creation of an Apache-hosted JIRA. The Dataflow > project > > is > > > > currently using both a public GitHub issue tracker and internal > Google > > > issue > > > > tracking. We will migrate and combine from these two sources to the > > > Apache > > > > JIRA. > > > > > > > > * Jira ID: BEAM > > > > > > > > == Initial Committers == > > > > > > > > * Aljoscha Krettek [aljos...@apache.org] > > > > * Amit Sela [amitsel...@gmail.com] > > > > * Ben Chambers [bchamb...@google.com] > > > > * Craig Chambers [chamb...@google.com] > > > > * Dan Halperin [dhalp...@google.com] > > > > * Davor Bonaci [da...@google.com] > > > > * Frances Perry [f...@google.com] > > > > * James Malone [jamesmal...@google.com] > > > > * Jean-Baptiste Onofré [jbono...@apache.org] > > > > * Josh Wills [jwi...@apache.org] > > > > * Kostas Tzoumas [kos...@data-artisans.com] > > > > * Kenneth Knowles [k...@google.com] > > > > * Luke Cwik [lc...@google.com] > > > > * Maximilian Michels [m...@apache.org] > > > > * Stephan Ewen [step...@data-artisans.com] > > > > * Tom White [t...@cloudera.com] > > > > * Tyler Akidau [taki...@google.com] > > > > * Robert Bradshaw [rober...@google.com] > > > > > > > > == Additional Interested Contributors == > > > > > > > > * Debo Dutta [dedu...@cisco.com] > > > > * Henry Saputra [hsapu...@apache.org] > > > > * Taylor Goetz [ptgo...@gmail.com] > > > > * James Carman [ja...@carmanconsulting.com] > > > > * Joe Witt [joew...@apache.org] > > > > * Vaibhav Gumashta [vgumas...@hortonworks.com] > > > > * Prasanth Jayachandran [pjayachand...@hortonworks.com] > > > > * Johan Edstrom [seij...@gmail.com] > > > > * Hugo Louro [hmclo...@gmail.com] > > > > * Krzysztof Sobkowiak [krzys.sobkow...@gmail.com] > > > > * Jeff Genender [jgenen...@apache.org] > > > > * Edward J. Yoon [edward.y...@samsung.com] > > > > * Hao Chen [h...@apache.org] > > > > * Byung-Gon Chun [bgc...@gmail.com] > > > > * Charitha Elvitigala [charit...@apache.org] > > > > * Alexander Bezzubov [b...@apache.org] > > > > * Tsuyoshi Ozawa [oz...@apache.org] > > > > * Mayank Bansal [maban...@gmail.com] > > > > * Supun Kamburugamuve [su...@apache.org] > > > > * Matthias Wessendorf [mat...@apache.org] > > > > * Felix Cheung [felixche...@apache.org] > > > > * Ajay Yadava [ajay.ya...@inmobi.com] > > > > * Liang Chen [chenliang...@huawei.com] > > > > * Renaud Richardet [renaud (at) apache (dot) org] > > > > * Bakey Pan [bakey1...@gmail.com] > > > > * Andreas Neumann [a...@apache.org] > > > > * Suresh Marru [sma...@apache.org] > > > > * Hadrian Zbarcea [hzbar...@gmail.com] > > > > > > > > == Affiliations == > > > > > > > > The initial committers are from six organizations. Google developed > > > Dataflow > > > > and the Dataflow SDK, data Artisans developed the Flink runner, and > > > Cloudera > > > > (Labs) developed the Spark runner. > > > > > > > > * Cloudera > > > > * Tom White > > > > * Data Artisans > > > > * Aljoscha Krettek > > > > * Kostas Tzoumas > > > > * Maximilian Michels > > > > * Stephan Ewen > > > > * Google > > > > * Ben Chambers > > > > * Dan Halperin > > > > * Davor Bonaci > > > > * Frances Perry > > > > * James Malone > > > > * Kenneth Knowles > > > > * Luke Cwik > > > > * Tyler Akidau > > > > * Robert Bradshaw > > > > * PayPal > > > > * Amit Sela > > > > * Slack > > > > * Josh Wills > > > > * Talend > > > > * Jean-Baptiste Onofré > > > > > > > > == Sponsors == > > > > > > > > === Champion === > > > > > > > > * Jean-Baptiste Onofre [jbono...@apache.org] > > > > > > > > === Nominated Mentors === > > > > > > > > * Jean-Baptiste Onofre [jbono...@apache.org] > > > > * Jim Jagielski [j...@apache.org] > > > > * Venkatesh Seetharam [venkat...@apache.org] > > > > * Bertrand Delacretaz [bdelacre...@apache.org] > > > > * Ted Dunning [tdunn...@apache.org] > > > > > > > > === Sponsoring Entity === > > > > > > > > The Apache Incubator > > > > ---- > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > > >
