+1 (non-binding) On Thu, Jan 28, 2016 at 3:54 PM, Joe Witt <joe.w...@gmail.com> wrote:
> +1 (non-binding) > > On Thu, Jan 28, 2016 at 9:48 AM, Sergio Fernández <wik...@apache.org> > wrote: > > +1 (binding) > > > > On Thu, Jan 28, 2016 at 3:28 PM, Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > >> Hi, > >> > >> the Beam proposal (initially Dataflow) was proposed last week. > >> > >> The complete discussion thread is available here: > >> > >> > >> > http://mail-archives.apache.org/mod_mbox/incubator-general/201601.mbox/%3CCA%2B%3DKJmvj4wyosNTXVpnsH8PhS7jEyzkZngc682rGgZ3p28L42Q%40mail.gmail.com%3E > >> > >> As reminder the BeamProposal is here: > >> > >> https://wiki.apache.org/incubator/BeamProposal > >> > >> Regarding all the great feedbacks we received on the mailing list, we > >> think it's time to call a vote to accept Beam into the Incubator. > >> > >> Please cast your vote to: > >> [] +1 - accept Apache Beam as a new incubating project > >> [] 0 - not sure > >> [] -1 - do not accept the Apache Beam project (because: ...) > >> > >> Thanks, > >> Regards > >> JB > >> ---- > >> ## page was renamed from DataflowProposal > >> = Apache Beam = > >> > >> == Abstract == > >> > >> Apache Beam is an open source, unified model and set of > language-specific > >> SDKs for defining and executing data processing workflows, and also data > >> ingestion and integration flows, supporting Enterprise Integration > Patterns > >> (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify > >> the mechanics of large-scale batch and streaming data processing and can > >> run on a number of runtimes like Apache Flink, Apache Spark, and Google > >> Cloud Dataflow (a cloud service). Beam also brings DSL in different > >> languages, allowing users to easily implement their data integration > >> processes. > >> > >> == Proposal == > >> > >> Beam is a simple, flexible, and powerful system for distributed data > >> processing at any scale. Beam provides a unified programming model, a > >> software development kit to define and construct data processing > pipelines, > >> and runners to execute Beam pipelines in several runtime engines, like > >> Apache Spark, Apache Flink, or Google Cloud Dataflow. Beam can be used > for > >> a variety of streaming or batch data processing goals including ETL, > stream > >> analysis, and aggregate computation. The underlying programming model > for > >> Beam provides MapReduce-like parallelism, combined with support for > >> powerful data windowing, and fine-grained correctness control. > >> > >> == Background == > >> > >> Beam started as a set of Google projects (Google Cloud Dataflow) focused > >> on making data processing easier, faster, and less costly. The Beam > model > >> is a successor to MapReduce, FlumeJava, and Millwheel inside Google and > is > >> focused on providing a unified solution for batch and stream processing. > >> These projects on which Beam is based have been published in several > papers > >> made available to the public: > >> > >> * MapReduce - http://research.google.com/archive/mapreduce.html > >> * Dataflow model - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf > >> * FlumeJava - http://research.google.com/pubs/pub35650.html > >> * MillWheel - http://research.google.com/pubs/pub41378.html > >> > >> Beam was designed from the start to provide a portable programming > layer. > >> When you define a data processing pipeline with the Beam model, you are > >> creating a job which is capable of being processed by any number of Beam > >> processing engines. Several engines have been developed to run Beam > >> pipelines in other open source runtimes, including a Beam runner for > Apache > >> Flink and Apache Spark. There is also a “direct runner”, for execution > on > >> the developer machine (mainly for dev/debug purposes). Another runner > >> allows a Beam program to run on a managed service, Google Cloud > Dataflow, > >> in Google Cloud Platform. The Dataflow Java SDK is already available on > >> GitHub, and independent from the Google Cloud Dataflow service. Another > >> Python SDK is currently in active development. > >> > >> In this proposal, the Beam SDKs, model, and a set of runners will be > >> submitted as an OSS project under the ASF. The runners which are a part > of > >> this proposal include those for Spark (from Cloudera), Flink (from data > >> Artisans), and local development (from Google); the Google Cloud > Dataflow > >> service runner is not included in this proposal. Further references to > Beam > >> will refer to the Dataflow model, SDKs, and runners which are a part of > >> this proposal (Apache Beam) only. The initial submission will contain > the > >> already-released Java SDK; Google intends to submit the Python SDK > later in > >> the incubation process. The Google Cloud Dataflow service will continue > to > >> be one of many runners for Beam, built on Google Cloud Platform, to run > >> Beam pipelines. Necessarily, Cloud Dataflow will develop against the > Apache > >> project additions, updates, and changes. Google Cloud Dataflow will > become > >> one user of Apache Beam and will participate in the project openly and > >> publicly. > >> > >> The Beam programming model has been designed with simplicity, > scalability, > >> and speed as key tenants. In the Beam model, you only need to think > about > >> four top-level concepts when constructing your data processing job: > >> > >> * Pipelines - The data processing job made of a series of computations > >> including input, processing, and output > >> * PCollections - Bounded (or unbounded) datasets which represent the > >> input, intermediate and output data in pipelines > >> * PTransforms - A data processing step in a pipeline in which one or > more > >> PCollections are an input and output > >> * I/O Sources and Sinks - APIs for reading and writing data which are > the > >> roots and endpoints of the pipeline > >> > >> == Rationale == > >> > >> With Google Dataflow, Google intended to develop a framework which > allowed > >> developers to be maximally productive in defining the processing, and > then > >> be able to execute the program at various levels of > >> latency/cost/completeness without re-architecting or re-writing it. This > >> goal was informed by Google’s past experience developing several > models, > >> frameworks, and tools useful for large-scale and distributed data > >> processing. While Google has previously published papers describing > some of > >> its technologies, Google decided to take a different approach with > >> Dataflow. Google open-sourced the SDK and model alongside > commercialization > >> of the idea and ahead of publishing papers on the topic. As a result, a > >> number of open source runtimes exist for Dataflow, such as the Apache > Flink > >> and Apache Spark runners. > >> > >> We believe that submitting Beam as an Apache project will provide an > >> immediate, worthwhile, and substantial contribution to the open source > >> community. As an incubating project, we believe Dataflow will have a > better > >> opportunity to provide a meaningful contribution to OSS and also > integrate > >> with other Apache projects. > >> > >> In the long term, we believe Beam can be a powerful abstraction layer > for > >> data processing. By providing an abstraction layer for data pipelines > and > >> processing, data workflows can be increasingly portable, resilient to > >> breaking changes in tooling, and compatible across many execution > engines, > >> runtimes, and open source projects. > >> > >> == Initial Goals == > >> > >> We are breaking our initial goals into immediate (< 2 months), > short-term > >> (2-4 months), and intermediate-term (> 4 months). > >> > >> Our immediate goals include the following: > >> > >> * Plan for reconciling the Dataflow Java SDK and various runners into > one > >> project > >> * Plan for refactoring the existing Java SDK for better extensibility > by > >> SDK and runner writers > >> * Validating all dependencies are ASL 2.0 or compatible > >> * Understanding and adapting to the Apache development process > >> > >> Our short-term goals include: > >> > >> * Moving the newly-merged lists, and build utilities to Apache > >> * Start refactoring codebase and move code to Apache Git repo > >> * Continue development of new features, functions, and fixes in the > >> Dataflow Java SDK, and Dataflow runners > >> * Cleaning up the Dataflow SDK sources and crafting a roadmap and plan > >> for how to include new major ideas, modules, and runtimes > >> * Establishment of easy and clear build/test framework for Dataflow and > >> associated runtimes; creation of testing, rollback, and validation > policy > >> * Analysis and design for work needed to make Beam a better data > >> processing abstraction layer for multiple open source frameworks and > >> environments > >> > >> Finally, we have a number of intermediate-term goals: > >> > >> * Roadmapping, planning, and execution of integrations with other OSS > and > >> non-OSS projects/products > >> * Inclusion of additional SDK for Python, which is under active > >> development > >> > >> == Current Status == > >> > >> === Meritocracy === > >> > >> Dataflow was initially developed based on ideas from many employees > within > >> Google. As an ASL OSS project on GitHub, the Dataflow SDK has received > >> contributions from data Artisans, Cloudera Labs, and other individual > >> developers. As a project under incubation, we are committed to expanding > >> our effort to build an environment which supports a meritocracy. We are > >> focused on engaging the community and other related projects for support > >> and contributions. Moreover, we are committed to ensure contributors and > >> committers to Dataflow come from a broad mix of organizations through a > >> merit-based decision process during incubation. We believe strongly in > the > >> Beam model and are committed to growing an inclusive community of Beam > >> contributors. > >> > >> === Community === > >> > >> The core of the Dataflow Java SDK has been developed by Google for use > >> with Google Cloud Dataflow. Google has active community engagement in > the > >> SDK GitHub repository ( > >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK), on Stack > >> Overflow ( > http://stackoverflow.com/questions/tagged/google-cloud-dataflow) > >> and has had contributions from a number of organizations and > indivuduals. > >> > >> Everyday, Cloud Dataflow is actively used by a number of organizations > and > >> institutions for batch and stream processing of data. We believe > acceptance > >> will allow us to consolidate existing Dataflow-related work, grow the > >> Dataflow community, and deepen connections between Dataflow and other > open > >> source projects. > >> > >> === Core Developers === > >> > >> The core developers for Dataflow and the Dataflow runners are: > >> > >> * Frances Perry > >> * Tyler Akidau > >> * Davor Bonaci > >> * Luke Cwik > >> * Ben Chambers > >> * Kenn Knowles > >> * Dan Halperin > >> * Daniel Mills > >> * Mark Shields > >> * Craig Chambers > >> * Maximilian Michels > >> * Tom White > >> * Josh Wills > >> * Robert Bradshaw > >> > >> === Alignment === > >> > >> The Beam SDK can be used to create Beam pipelines which can be executed > on > >> Apache Spark or Apache Flink. Beam is also related to other Apache > >> projects, such as Apache Crunch. We plan on expanding functionality for > >> Beam runners, support for additional domain specific languages, and > >> increased portability so Beam is a powerful abstraction layer for data > >> processing. > >> > >> == Known Risks == > >> > >> === Orphaned Products === > >> > >> The Dataflow SDK is presently used by several organizations, from small > >> startups to Fortune 100 companies, to construct production pipelines > which > >> are executed in Google Cloud Dataflow. Google has a long-term > commitment to > >> advance the Dataflow SDK; moreover, Dataflow is seeing increasing > interest, > >> development, and adoption from organizations outside of Google. > >> > >> === Inexperience with Open Source === > >> > >> Google believes strongly in open source and the exchange of information > to > >> advance new ideas and work. Examples of this commitment are active OSS > >> projects such as Chromium (https://www.chromium.org) and Kubernetes ( > >> http://kubernetes.io/). With Dataflow, we have tried to be increasingly > >> open and forward-looking; we have published a paper in the VLDB > conference > >> describing the Dataflow model ( > >> http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to > >> release the Dataflow SDK as open source software with the launch of > Cloud > >> Dataflow. Our submission to the Apache Software Foundation is a logical > >> extension of our commitment to open source software. > >> > >> === Homogeneous Developers === > >> > >> The majority of committers in this proposal belong to Google due to the > >> fact that Dataflow has emerged from several internal Google projects. > This > >> proposal also includes committers outside of Google who are actively > >> involved with other Apache projects, such as Hadoop, Flink, and Spark. > We > >> expect our entry into incubation will allow us to expand the number of > >> individuals and organizations participating in Dataflow development. > >> Additionally, separation of the Dataflow SDK from Google Cloud Dataflow > >> allows us to focus on the open source SDK and model and do what is best > for > >> this project. > >> > >> === Reliance on Salaried Developers === > >> > >> The Dataflow SDK and Dataflow runners have been developed primarily by > >> salaried developers supporting the Google Cloud Dataflow project. While > the > >> Dataflow SDK and Cloud Dataflow have been developed by different teams > (and > >> this proposal would reinforce that separation) we expect our initial > set of > >> developers will still primarily be salaried. Contribution has not been > >> exclusively from salaried developers, however. For example, the contrib > >> directory of the Dataflow SDK ( > >> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib > ) > >> contains items from free-time contributors. Moreover, seperate projects, > >> such as ScalaFlow (https://github.com/darkjh/scalaflow) have been > created > >> around the Dataflow model and SDK. We expect our reliance on salaried > >> developers will decrease over time during incubation. > >> > >> === Relationship with other Apache products === > >> > >> Dataflow directly interoperates with or utilizes several existing Apache > >> projects. > >> > >> * Build > >> * Apache Maven > >> * Data I/O, Libraries > >> * Apache Avro > >> * Apache Commons > >> * Dataflow runners > >> * Apache Flink > >> * Apache Spark > >> > >> Beam when used in batch mode shares similarities with Apache Crunch; > >> however, Beam is focused on a model, SDK, and abstraction layer beyond > >> Spark and Hadoop (MapReduce.) One key goal of Beam is to provide an > >> intermediate abstraction layer which can easily be implemented and > utilized > >> across several different processing frameworks. > >> > >> === An excessive fascination with the Apache brand === > >> > >> With this proposal we are not seeking attention or publicity. Rather, we > >> firmly believe in the Beam model, SDK, and the ability to make Beam a > >> powerful yet simple framework for data processing. While the Dataflow > SDK > >> and model have been open source, we believe putting code on GitHub can > only > >> go so far. We see the Apache community, processes, and mission as > critical > >> for ensuring the Beam SDK and model are truly community-driven, > positively > >> impactful, and innovative open source software. While Google has taken a > >> number of steps to advance its various open source projects, we believe > >> Beam is a great fit for the Apache Software Foundation due to its focus > on > >> data processing and its relationships to existing ASF projects. > >> > >> == Documentation == > >> > >> The following documentation is relevant to this proposal. Relevant > portion > >> of the documentation will be contributed to the Apache Beam project. > >> > >> * Dataflow website: https://cloud.google.com/dataflow > >> * Dataflow programming model: > >> https://cloud.google.com/dataflow/model/programming-model > >> * Codebases > >> * Dataflow Java SDK: > >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK > >> * Flink Dataflow runner: > https://github.com/dataArtisans/flink-dataflow > >> * Spark Dataflow runner: https://github.com/cloudera/spark-dataflow > >> * Dataflow Java SDK issue tracker: > >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues > >> * google-cloud-dataflow tag on Stack Overflow: > >> http://stackoverflow.com/questions/tagged/google-cloud-dataflow > >> > >> == Initial Source == > >> > >> The initial source for Beam which we will submit to the Apache > Foundation > >> will include several related projects which are currently hosted on the > >> GitHub repositories: > >> > >> * Dataflow Java SDK ( > >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK) > >> * Flink Dataflow runner ( > https://github.com/dataArtisans/flink-dataflow) > >> * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow) > >> > >> These projects have always been Apache 2.0 licensed. We intend to bundle > >> all of these repositories since they are all complimentary and should be > >> maintained in one project. Prior to our submission, we will combine all > of > >> these projects into a new git repository. > >> > >> == Source and Intellectual Property Submission Plan == > >> > >> The source for the Dataflow SDK and the three runners (Spark, Flink, > >> Google Cloud Dataflow) are already licensed under an Apache 2 license. > >> > >> * Dataflow SDK - > >> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE > >> * Flink runner - > >> https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE > >> * Spark runner - > >> https://github.com/cloudera/spark-dataflow/blob/master/LICENSE > >> > >> Contributors to the Dataflow SDK have also signed the Google Individual > >> Contributor License Agreement ( > >> https://cla.developers.google.com/about/google-individual) in order to > >> contribute to the project. > >> > >> With respect to trademark rights, Google does not hold a trademark on > the > >> phrase “Dataflow.” Based on feedback and guidance we receive during the > >> incubation process, we are open to renaming the project if necessary for > >> trademark or other concerns. > >> > >> == External Dependencies == > >> > >> All external dependencies are licensed under an Apache 2.0 or > >> Apache-compatible license. As we grow the Beam community we will > configure > >> our build process to require and validate all contributions and > >> dependencies are licensed under the Apache 2.0 license or are under an > >> Apache-compatible license. > >> > >> == Required Resources == > >> > >> === Mailing Lists === > >> > >> We currently use a mix of mailing lists. We will migrate our existing > >> mailing lists to the following: > >> > >> * d...@beam.incubator.apache.org > >> * u...@beam.incubator.apache.org > >> * priv...@beam.incubator.apache.org > >> * comm...@beam.incubator.apache.org > >> > >> === Source Control === > >> > >> The Dataflow team currently uses Git and would like to continue to do > so. > >> We request a Git repository for Beam with mirroring to GitHub enabled. > >> > >> * https://git-wip-us.apache.org/repos/asf/incubator-beam.git > >> > >> === Issue Tracking === > >> > >> We request the creation of an Apache-hosted JIRA. The Dataflow project > is > >> currently using both a public GitHub issue tracker and internal Google > >> issue tracking. We will migrate and combine from these two sources to > the > >> Apache JIRA. > >> > >> * Jira ID: BEAM > >> > >> == Initial Committers == > >> > >> * Aljoscha Krettek [aljos...@apache.org] > >> * Amit Sela [amitsel...@gmail.com] > >> * Ben Chambers [bchamb...@google.com] > >> * Craig Chambers [chamb...@google.com] > >> * Dan Halperin [dhalp...@google.com] > >> * Davor Bonaci [da...@google.com] > >> * Frances Perry [f...@google.com] > >> * James Malone [jamesmal...@google.com] > >> * Jean-Baptiste Onofré [jbono...@apache.org] > >> * Josh Wills [jwi...@apache.org] > >> * Kostas Tzoumas [kos...@data-artisans.com] > >> * Kenneth Knowles [k...@google.com] > >> * Luke Cwik [lc...@google.com] > >> * Maximilian Michels [m...@apache.org] > >> * Stephan Ewen [step...@data-artisans.com] > >> * Tom White [t...@cloudera.com] > >> * Tyler Akidau [taki...@google.com] > >> * Robert Bradshaw [rober...@google.com] > >> > >> == Additional Interested Contributors == > >> > >> * Debo Dutta [dedu...@cisco.com] > >> * Henry Saputra [hsapu...@apache.org] > >> * Taylor Goetz [ptgo...@gmail.com] > >> * James Carman [ja...@carmanconsulting.com] > >> * Joe Witt [joew...@apache.org] > >> * Vaibhav Gumashta [vgumas...@hortonworks.com] > >> * Prasanth Jayachandran [pjayachand...@hortonworks.com] > >> * Johan Edstrom [seij...@gmail.com] > >> * Hugo Louro [hmclo...@gmail.com] > >> * Krzysztof Sobkowiak [krzys.sobkow...@gmail.com] > >> * Jeff Genender [jgenen...@apache.org] > >> * Edward J. Yoon [edward.y...@samsung.com] > >> * Hao Chen [h...@apache.org] > >> * Byung-Gon Chun [bgc...@gmail.com] > >> * Charitha Elvitigala [charit...@apache.org] > >> * Alexander Bezzubov [b...@apache.org] > >> * Tsuyoshi Ozawa [oz...@apache.org] > >> * Mayank Bansal [maban...@gmail.com] > >> * Supun Kamburugamuve [su...@apache.org] > >> * Matthias Wessendorf [mat...@apache.org] > >> * Felix Cheung [felixche...@apache.org] > >> * Ajay Yadava [ajay.ya...@inmobi.com] > >> * Liang Chen [chenliang...@huawei.com] > >> * Renaud Richardet [renaud (at) apache (dot) org] > >> * Bakey Pan [bakey1...@gmail.com] > >> * Andreas Neumann [a...@apache.org] > >> * Suresh Marru [sma...@apache.org] > >> * Hadrian Zbarcea [hzbar...@gmail.com] > >> > >> == Affiliations == > >> > >> The initial committers are from six organizations. Google developed > >> Dataflow and the Dataflow SDK, data Artisans developed the Flink runner, > >> and Cloudera (Labs) developed the Spark runner. > >> > >> * Cloudera > >> * Tom White > >> * Data Artisans > >> * Aljoscha Krettek > >> * Kostas Tzoumas > >> * Maximilian Michels > >> * Stephan Ewen > >> * Google > >> * Ben Chambers > >> * Dan Halperin > >> * Davor Bonaci > >> * Frances Perry > >> * James Malone > >> * Kenneth Knowles > >> * Luke Cwik > >> * Tyler Akidau > >> * Robert Bradshaw > >> * PayPal > >> * Amit Sela > >> * Slack > >> * Josh Wills > >> * Talend > >> * Jean-Baptiste Onofré > >> > >> == Sponsors == > >> > >> === Champion === > >> > >> * Jean-Baptiste Onofre [jbono...@apache.org] > >> > >> === Nominated Mentors === > >> > >> * Jean-Baptiste Onofre [jbono...@apache.org] > >> * Jim Jagielski [j...@apache.org] > >> * Venkatesh Seetharam [venkat...@apache.org] > >> * Bertrand Delacretaz [bdelacre...@apache.org] > >> * Ted Dunning [tdunn...@apache.org] > >> > >> === Sponsoring Entity === > >> > >> The Apache Incubator > >> ---- > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> For additional commands, e-mail: general-h...@incubator.apache.org > >> > >> > > > > > > -- > > Sergio Fernández > > Partner Technology Manager > > Redlink GmbH > > m: +43 6602747925 > > e: sergio.fernan...@redlink.co > > w: http://redlink.co > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
