+1 (non-binding) > On 28 Jan 2016, at 15:59, Alexander Bezzubov <b...@apache.org> wrote: > > +1 (non-binding) > > On Thu, Jan 28, 2016 at 3:54 PM, Joe Witt <joe.w...@gmail.com> wrote: > >> +1 (non-binding) >> >> On Thu, Jan 28, 2016 at 9:48 AM, Sergio Fernández <wik...@apache.org> >> wrote: >>> +1 (binding) >>> >>> On Thu, Jan 28, 2016 at 3:28 PM, Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>>> Hi, >>>> >>>> the Beam proposal (initially Dataflow) was proposed last week. >>>> >>>> The complete discussion thread is available here: >>>> >>>> >>>> >> http://mail-archives.apache.org/mod_mbox/incubator-general/201601.mbox/%3CCA%2B%3DKJmvj4wyosNTXVpnsH8PhS7jEyzkZngc682rGgZ3p28L42Q%40mail.gmail.com%3E >>>> >>>> As reminder the BeamProposal is here: >>>> >>>> https://wiki.apache.org/incubator/BeamProposal >>>> >>>> Regarding all the great feedbacks we received on the mailing list, we >>>> think it's time to call a vote to accept Beam into the Incubator. >>>> >>>> Please cast your vote to: >>>> [] +1 - accept Apache Beam as a new incubating project >>>> [] 0 - not sure >>>> [] -1 - do not accept the Apache Beam project (because: ...) >>>> >>>> Thanks, >>>> Regards >>>> JB >>>> ---- >>>> ## page was renamed from DataflowProposal >>>> = Apache Beam = >>>> >>>> == Abstract == >>>> >>>> Apache Beam is an open source, unified model and set of >> language-specific >>>> SDKs for defining and executing data processing workflows, and also data >>>> ingestion and integration flows, supporting Enterprise Integration >> Patterns >>>> (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify >>>> the mechanics of large-scale batch and streaming data processing and can >>>> run on a number of runtimes like Apache Flink, Apache Spark, and Google >>>> Cloud Dataflow (a cloud service). Beam also brings DSL in different >>>> languages, allowing users to easily implement their data integration >>>> processes. >>>> >>>> == Proposal == >>>> >>>> Beam is a simple, flexible, and powerful system for distributed data >>>> processing at any scale. Beam provides a unified programming model, a >>>> software development kit to define and construct data processing >> pipelines, >>>> and runners to execute Beam pipelines in several runtime engines, like >>>> Apache Spark, Apache Flink, or Google Cloud Dataflow. Beam can be used >> for >>>> a variety of streaming or batch data processing goals including ETL, >> stream >>>> analysis, and aggregate computation. The underlying programming model >> for >>>> Beam provides MapReduce-like parallelism, combined with support for >>>> powerful data windowing, and fine-grained correctness control. >>>> >>>> == Background == >>>> >>>> Beam started as a set of Google projects (Google Cloud Dataflow) focused >>>> on making data processing easier, faster, and less costly. The Beam >> model >>>> is a successor to MapReduce, FlumeJava, and Millwheel inside Google and >> is >>>> focused on providing a unified solution for batch and stream processing. >>>> These projects on which Beam is based have been published in several >> papers >>>> made available to the public: >>>> >>>> * MapReduce - http://research.google.com/archive/mapreduce.html >>>> * Dataflow model - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf >>>> * FlumeJava - http://research.google.com/pubs/pub35650.html >>>> * MillWheel - http://research.google.com/pubs/pub41378.html >>>> >>>> Beam was designed from the start to provide a portable programming >> layer. >>>> When you define a data processing pipeline with the Beam model, you are >>>> creating a job which is capable of being processed by any number of Beam >>>> processing engines. Several engines have been developed to run Beam >>>> pipelines in other open source runtimes, including a Beam runner for >> Apache >>>> Flink and Apache Spark. There is also a “direct runner”, for execution >> on >>>> the developer machine (mainly for dev/debug purposes). Another runner >>>> allows a Beam program to run on a managed service, Google Cloud >> Dataflow, >>>> in Google Cloud Platform. The Dataflow Java SDK is already available on >>>> GitHub, and independent from the Google Cloud Dataflow service. Another >>>> Python SDK is currently in active development. >>>> >>>> In this proposal, the Beam SDKs, model, and a set of runners will be >>>> submitted as an OSS project under the ASF. The runners which are a part >> of >>>> this proposal include those for Spark (from Cloudera), Flink (from data >>>> Artisans), and local development (from Google); the Google Cloud >> Dataflow >>>> service runner is not included in this proposal. Further references to >> Beam >>>> will refer to the Dataflow model, SDKs, and runners which are a part of >>>> this proposal (Apache Beam) only. The initial submission will contain >> the >>>> already-released Java SDK; Google intends to submit the Python SDK >> later in >>>> the incubation process. The Google Cloud Dataflow service will continue >> to >>>> be one of many runners for Beam, built on Google Cloud Platform, to run >>>> Beam pipelines. Necessarily, Cloud Dataflow will develop against the >> Apache >>>> project additions, updates, and changes. Google Cloud Dataflow will >> become >>>> one user of Apache Beam and will participate in the project openly and >>>> publicly. >>>> >>>> The Beam programming model has been designed with simplicity, >> scalability, >>>> and speed as key tenants. In the Beam model, you only need to think >> about >>>> four top-level concepts when constructing your data processing job: >>>> >>>> * Pipelines - The data processing job made of a series of computations >>>> including input, processing, and output >>>> * PCollections - Bounded (or unbounded) datasets which represent the >>>> input, intermediate and output data in pipelines >>>> * PTransforms - A data processing step in a pipeline in which one or >> more >>>> PCollections are an input and output >>>> * I/O Sources and Sinks - APIs for reading and writing data which are >> the >>>> roots and endpoints of the pipeline >>>> >>>> == Rationale == >>>> >>>> With Google Dataflow, Google intended to develop a framework which >> allowed >>>> developers to be maximally productive in defining the processing, and >> then >>>> be able to execute the program at various levels of >>>> latency/cost/completeness without re-architecting or re-writing it. This >>>> goal was informed by Google’s past experience developing several >> models, >>>> frameworks, and tools useful for large-scale and distributed data >>>> processing. While Google has previously published papers describing >> some of >>>> its technologies, Google decided to take a different approach with >>>> Dataflow. Google open-sourced the SDK and model alongside >> commercialization >>>> of the idea and ahead of publishing papers on the topic. As a result, a >>>> number of open source runtimes exist for Dataflow, such as the Apache >> Flink >>>> and Apache Spark runners. >>>> >>>> We believe that submitting Beam as an Apache project will provide an >>>> immediate, worthwhile, and substantial contribution to the open source >>>> community. As an incubating project, we believe Dataflow will have a >> better >>>> opportunity to provide a meaningful contribution to OSS and also >> integrate >>>> with other Apache projects. >>>> >>>> In the long term, we believe Beam can be a powerful abstraction layer >> for >>>> data processing. By providing an abstraction layer for data pipelines >> and >>>> processing, data workflows can be increasingly portable, resilient to >>>> breaking changes in tooling, and compatible across many execution >> engines, >>>> runtimes, and open source projects. >>>> >>>> == Initial Goals == >>>> >>>> We are breaking our initial goals into immediate (< 2 months), >> short-term >>>> (2-4 months), and intermediate-term (> 4 months). >>>> >>>> Our immediate goals include the following: >>>> >>>> * Plan for reconciling the Dataflow Java SDK and various runners into >> one >>>> project >>>> * Plan for refactoring the existing Java SDK for better extensibility >> by >>>> SDK and runner writers >>>> * Validating all dependencies are ASL 2.0 or compatible >>>> * Understanding and adapting to the Apache development process >>>> >>>> Our short-term goals include: >>>> >>>> * Moving the newly-merged lists, and build utilities to Apache >>>> * Start refactoring codebase and move code to Apache Git repo >>>> * Continue development of new features, functions, and fixes in the >>>> Dataflow Java SDK, and Dataflow runners >>>> * Cleaning up the Dataflow SDK sources and crafting a roadmap and plan >>>> for how to include new major ideas, modules, and runtimes >>>> * Establishment of easy and clear build/test framework for Dataflow and >>>> associated runtimes; creation of testing, rollback, and validation >> policy >>>> * Analysis and design for work needed to make Beam a better data >>>> processing abstraction layer for multiple open source frameworks and >>>> environments >>>> >>>> Finally, we have a number of intermediate-term goals: >>>> >>>> * Roadmapping, planning, and execution of integrations with other OSS >> and >>>> non-OSS projects/products >>>> * Inclusion of additional SDK for Python, which is under active >>>> development >>>> >>>> == Current Status == >>>> >>>> === Meritocracy === >>>> >>>> Dataflow was initially developed based on ideas from many employees >> within >>>> Google. As an ASL OSS project on GitHub, the Dataflow SDK has received >>>> contributions from data Artisans, Cloudera Labs, and other individual >>>> developers. As a project under incubation, we are committed to expanding >>>> our effort to build an environment which supports a meritocracy. We are >>>> focused on engaging the community and other related projects for support >>>> and contributions. Moreover, we are committed to ensure contributors and >>>> committers to Dataflow come from a broad mix of organizations through a >>>> merit-based decision process during incubation. We believe strongly in >> the >>>> Beam model and are committed to growing an inclusive community of Beam >>>> contributors. >>>> >>>> === Community === >>>> >>>> The core of the Dataflow Java SDK has been developed by Google for use >>>> with Google Cloud Dataflow. Google has active community engagement in >> the >>>> SDK GitHub repository ( >>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK), on Stack >>>> Overflow ( >> http://stackoverflow.com/questions/tagged/google-cloud-dataflow) >>>> and has had contributions from a number of organizations and >> indivuduals. >>>> >>>> Everyday, Cloud Dataflow is actively used by a number of organizations >> and >>>> institutions for batch and stream processing of data. We believe >> acceptance >>>> will allow us to consolidate existing Dataflow-related work, grow the >>>> Dataflow community, and deepen connections between Dataflow and other >> open >>>> source projects. >>>> >>>> === Core Developers === >>>> >>>> The core developers for Dataflow and the Dataflow runners are: >>>> >>>> * Frances Perry >>>> * Tyler Akidau >>>> * Davor Bonaci >>>> * Luke Cwik >>>> * Ben Chambers >>>> * Kenn Knowles >>>> * Dan Halperin >>>> * Daniel Mills >>>> * Mark Shields >>>> * Craig Chambers >>>> * Maximilian Michels >>>> * Tom White >>>> * Josh Wills >>>> * Robert Bradshaw >>>> >>>> === Alignment === >>>> >>>> The Beam SDK can be used to create Beam pipelines which can be executed >> on >>>> Apache Spark or Apache Flink. Beam is also related to other Apache >>>> projects, such as Apache Crunch. We plan on expanding functionality for >>>> Beam runners, support for additional domain specific languages, and >>>> increased portability so Beam is a powerful abstraction layer for data >>>> processing. >>>> >>>> == Known Risks == >>>> >>>> === Orphaned Products === >>>> >>>> The Dataflow SDK is presently used by several organizations, from small >>>> startups to Fortune 100 companies, to construct production pipelines >> which >>>> are executed in Google Cloud Dataflow. Google has a long-term >> commitment to >>>> advance the Dataflow SDK; moreover, Dataflow is seeing increasing >> interest, >>>> development, and adoption from organizations outside of Google. >>>> >>>> === Inexperience with Open Source === >>>> >>>> Google believes strongly in open source and the exchange of information >> to >>>> advance new ideas and work. Examples of this commitment are active OSS >>>> projects such as Chromium (https://www.chromium.org) and Kubernetes ( >>>> http://kubernetes.io/). With Dataflow, we have tried to be increasingly >>>> open and forward-looking; we have published a paper in the VLDB >> conference >>>> describing the Dataflow model ( >>>> http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to >>>> release the Dataflow SDK as open source software with the launch of >> Cloud >>>> Dataflow. Our submission to the Apache Software Foundation is a logical >>>> extension of our commitment to open source software. >>>> >>>> === Homogeneous Developers === >>>> >>>> The majority of committers in this proposal belong to Google due to the >>>> fact that Dataflow has emerged from several internal Google projects. >> This >>>> proposal also includes committers outside of Google who are actively >>>> involved with other Apache projects, such as Hadoop, Flink, and Spark. >> We >>>> expect our entry into incubation will allow us to expand the number of >>>> individuals and organizations participating in Dataflow development. >>>> Additionally, separation of the Dataflow SDK from Google Cloud Dataflow >>>> allows us to focus on the open source SDK and model and do what is best >> for >>>> this project. >>>> >>>> === Reliance on Salaried Developers === >>>> >>>> The Dataflow SDK and Dataflow runners have been developed primarily by >>>> salaried developers supporting the Google Cloud Dataflow project. While >> the >>>> Dataflow SDK and Cloud Dataflow have been developed by different teams >> (and >>>> this proposal would reinforce that separation) we expect our initial >> set of >>>> developers will still primarily be salaried. Contribution has not been >>>> exclusively from salaried developers, however. For example, the contrib >>>> directory of the Dataflow SDK ( >>>> >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib >> ) >>>> contains items from free-time contributors. Moreover, seperate projects, >>>> such as ScalaFlow (https://github.com/darkjh/scalaflow) have been >> created >>>> around the Dataflow model and SDK. We expect our reliance on salaried >>>> developers will decrease over time during incubation. >>>> >>>> === Relationship with other Apache products === >>>> >>>> Dataflow directly interoperates with or utilizes several existing Apache >>>> projects. >>>> >>>> * Build >>>> * Apache Maven >>>> * Data I/O, Libraries >>>> * Apache Avro >>>> * Apache Commons >>>> * Dataflow runners >>>> * Apache Flink >>>> * Apache Spark >>>> >>>> Beam when used in batch mode shares similarities with Apache Crunch; >>>> however, Beam is focused on a model, SDK, and abstraction layer beyond >>>> Spark and Hadoop (MapReduce.) One key goal of Beam is to provide an >>>> intermediate abstraction layer which can easily be implemented and >> utilized >>>> across several different processing frameworks. >>>> >>>> === An excessive fascination with the Apache brand === >>>> >>>> With this proposal we are not seeking attention or publicity. Rather, we >>>> firmly believe in the Beam model, SDK, and the ability to make Beam a >>>> powerful yet simple framework for data processing. While the Dataflow >> SDK >>>> and model have been open source, we believe putting code on GitHub can >> only >>>> go so far. We see the Apache community, processes, and mission as >> critical >>>> for ensuring the Beam SDK and model are truly community-driven, >> positively >>>> impactful, and innovative open source software. While Google has taken a >>>> number of steps to advance its various open source projects, we believe >>>> Beam is a great fit for the Apache Software Foundation due to its focus >> on >>>> data processing and its relationships to existing ASF projects. >>>> >>>> == Documentation == >>>> >>>> The following documentation is relevant to this proposal. Relevant >> portion >>>> of the documentation will be contributed to the Apache Beam project. >>>> >>>> * Dataflow website: https://cloud.google.com/dataflow >>>> * Dataflow programming model: >>>> https://cloud.google.com/dataflow/model/programming-model >>>> * Codebases >>>> * Dataflow Java SDK: >>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK >>>> * Flink Dataflow runner: >> https://github.com/dataArtisans/flink-dataflow >>>> * Spark Dataflow runner: https://github.com/cloudera/spark-dataflow >>>> * Dataflow Java SDK issue tracker: >>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues >>>> * google-cloud-dataflow tag on Stack Overflow: >>>> http://stackoverflow.com/questions/tagged/google-cloud-dataflow >>>> >>>> == Initial Source == >>>> >>>> The initial source for Beam which we will submit to the Apache >> Foundation >>>> will include several related projects which are currently hosted on the >>>> GitHub repositories: >>>> >>>> * Dataflow Java SDK ( >>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK) >>>> * Flink Dataflow runner ( >> https://github.com/dataArtisans/flink-dataflow) >>>> * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow) >>>> >>>> These projects have always been Apache 2.0 licensed. We intend to bundle >>>> all of these repositories since they are all complimentary and should be >>>> maintained in one project. Prior to our submission, we will combine all >> of >>>> these projects into a new git repository. >>>> >>>> == Source and Intellectual Property Submission Plan == >>>> >>>> The source for the Dataflow SDK and the three runners (Spark, Flink, >>>> Google Cloud Dataflow) are already licensed under an Apache 2 license. >>>> >>>> * Dataflow SDK - >>>> >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE >>>> * Flink runner - >>>> https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE >>>> * Spark runner - >>>> https://github.com/cloudera/spark-dataflow/blob/master/LICENSE >>>> >>>> Contributors to the Dataflow SDK have also signed the Google Individual >>>> Contributor License Agreement ( >>>> https://cla.developers.google.com/about/google-individual) in order to >>>> contribute to the project. >>>> >>>> With respect to trademark rights, Google does not hold a trademark on >> the >>>> phrase “Dataflow.” Based on feedback and guidance we receive during the >>>> incubation process, we are open to renaming the project if necessary for >>>> trademark or other concerns. >>>> >>>> == External Dependencies == >>>> >>>> All external dependencies are licensed under an Apache 2.0 or >>>> Apache-compatible license. As we grow the Beam community we will >> configure >>>> our build process to require and validate all contributions and >>>> dependencies are licensed under the Apache 2.0 license or are under an >>>> Apache-compatible license. >>>> >>>> == Required Resources == >>>> >>>> === Mailing Lists === >>>> >>>> We currently use a mix of mailing lists. We will migrate our existing >>>> mailing lists to the following: >>>> >>>> * d...@beam.incubator.apache.org >>>> * u...@beam.incubator.apache.org >>>> * priv...@beam.incubator.apache.org >>>> * comm...@beam.incubator.apache.org >>>> >>>> === Source Control === >>>> >>>> The Dataflow team currently uses Git and would like to continue to do >> so. >>>> We request a Git repository for Beam with mirroring to GitHub enabled. >>>> >>>> * https://git-wip-us.apache.org/repos/asf/incubator-beam.git >>>> >>>> === Issue Tracking === >>>> >>>> We request the creation of an Apache-hosted JIRA. The Dataflow project >> is >>>> currently using both a public GitHub issue tracker and internal Google >>>> issue tracking. We will migrate and combine from these two sources to >> the >>>> Apache JIRA. >>>> >>>> * Jira ID: BEAM >>>> >>>> == Initial Committers == >>>> >>>> * Aljoscha Krettek [aljos...@apache.org] >>>> * Amit Sela [amitsel...@gmail.com] >>>> * Ben Chambers [bchamb...@google.com] >>>> * Craig Chambers [chamb...@google.com] >>>> * Dan Halperin [dhalp...@google.com] >>>> * Davor Bonaci [da...@google.com] >>>> * Frances Perry [f...@google.com] >>>> * James Malone [jamesmal...@google.com] >>>> * Jean-Baptiste Onofré [jbono...@apache.org] >>>> * Josh Wills [jwi...@apache.org] >>>> * Kostas Tzoumas [kos...@data-artisans.com] >>>> * Kenneth Knowles [k...@google.com] >>>> * Luke Cwik [lc...@google.com] >>>> * Maximilian Michels [m...@apache.org] >>>> * Stephan Ewen [step...@data-artisans.com] >>>> * Tom White [t...@cloudera.com] >>>> * Tyler Akidau [taki...@google.com] >>>> * Robert Bradshaw [rober...@google.com] >>>> >>>> == Additional Interested Contributors == >>>> >>>> * Debo Dutta [dedu...@cisco.com] >>>> * Henry Saputra [hsapu...@apache.org] >>>> * Taylor Goetz [ptgo...@gmail.com] >>>> * James Carman [ja...@carmanconsulting.com] >>>> * Joe Witt [joew...@apache.org] >>>> * Vaibhav Gumashta [vgumas...@hortonworks.com] >>>> * Prasanth Jayachandran [pjayachand...@hortonworks.com] >>>> * Johan Edstrom [seij...@gmail.com] >>>> * Hugo Louro [hmclo...@gmail.com] >>>> * Krzysztof Sobkowiak [krzys.sobkow...@gmail.com] >>>> * Jeff Genender [jgenen...@apache.org] >>>> * Edward J. Yoon [edward.y...@samsung.com] >>>> * Hao Chen [h...@apache.org] >>>> * Byung-Gon Chun [bgc...@gmail.com] >>>> * Charitha Elvitigala [charit...@apache.org] >>>> * Alexander Bezzubov [b...@apache.org] >>>> * Tsuyoshi Ozawa [oz...@apache.org] >>>> * Mayank Bansal [maban...@gmail.com] >>>> * Supun Kamburugamuve [su...@apache.org] >>>> * Matthias Wessendorf [mat...@apache.org] >>>> * Felix Cheung [felixche...@apache.org] >>>> * Ajay Yadava [ajay.ya...@inmobi.com] >>>> * Liang Chen [chenliang...@huawei.com] >>>> * Renaud Richardet [renaud (at) apache (dot) org] >>>> * Bakey Pan [bakey1...@gmail.com] >>>> * Andreas Neumann [a...@apache.org] >>>> * Suresh Marru [sma...@apache.org] >>>> * Hadrian Zbarcea [hzbar...@gmail.com] >>>> >>>> == Affiliations == >>>> >>>> The initial committers are from six organizations. Google developed >>>> Dataflow and the Dataflow SDK, data Artisans developed the Flink runner, >>>> and Cloudera (Labs) developed the Spark runner. >>>> >>>> * Cloudera >>>> * Tom White >>>> * Data Artisans >>>> * Aljoscha Krettek >>>> * Kostas Tzoumas >>>> * Maximilian Michels >>>> * Stephan Ewen >>>> * Google >>>> * Ben Chambers >>>> * Dan Halperin >>>> * Davor Bonaci >>>> * Frances Perry >>>> * James Malone >>>> * Kenneth Knowles >>>> * Luke Cwik >>>> * Tyler Akidau >>>> * Robert Bradshaw >>>> * PayPal >>>> * Amit Sela >>>> * Slack >>>> * Josh Wills >>>> * Talend >>>> * Jean-Baptiste Onofré >>>> >>>> == Sponsors == >>>> >>>> === Champion === >>>> >>>> * Jean-Baptiste Onofre [jbono...@apache.org] >>>> >>>> === Nominated Mentors === >>>> >>>> * Jean-Baptiste Onofre [jbono...@apache.org] >>>> * Jim Jagielski [j...@apache.org] >>>> * Venkatesh Seetharam [venkat...@apache.org] >>>> * Bertrand Delacretaz [bdelacre...@apache.org] >>>> * Ted Dunning [tdunn...@apache.org] >>>> >>>> === Sponsoring Entity === >>>> >>>> The Apache Incubator >>>> ---- >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>> >>>> >>> >>> >>> -- >>> Sergio Fernández >>> Partner Technology Manager >>> Redlink GmbH >>> m: +43 6602747925 >>> e: sergio.fernan...@redlink.co >>> w: http://redlink.co >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >>
--------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
