Hi Jean, Nice Proposal.
I wanted to contribute to this project. Can you please add me too? Thanks a lot for the help Thanks, Mayank On Thu, Jan 21, 2016 at 8:07 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hey Alex, > > awesome: I added you on the proposal. > > Thanks, > Regards > JB > > > On 01/21/2016 05:03 PM, Alexander Bezzubov wrote: > >> Hi, >> >> it's great to see DataFlow becoming part to Apache ecosystem, thank you >> bringing it in. >> I would be happy to get involved and help. >> >> -- >> Alex >> >> On Thu, Jan 21, 2016 at 8:42 PM, Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >> Perfect: done, you are on the proposal. >>> >>> Thanks ! >>> Regards >>> JB >>> >>> >>> On 01/21/2016 11:55 AM, chatz wrote: >>> >>> Charitha Elvitigala >>>> >>>> On 21 January 2016 at 16:17, Jean-Baptiste Onofré <j...@nanthrax.net> >>>> wrote: >>>> >>>> Hi Chatz, >>>> >>>>> >>>>> sure, what name should I use on the proposal, Charitha ? >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> >>>>> On 01/21/2016 11:32 AM, chatz wrote: >>>>> >>>>> Hi Jean, >>>>> >>>>>> >>>>>> I’d be interested in contributing as well. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Chatz >>>>>> >>>>>> >>>>>> On 21 January 2016 at 14:22, Jean-Baptiste Onofré <j...@nanthrax.net> >>>>>> wrote: >>>>>> >>>>>> Sweet: you are on the proposal ;) >>>>>> >>>>>> >>>>>>> Thanks ! >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> >>>>>>> On 01/21/2016 08:55 AM, Byung-Gon Chun wrote: >>>>>>> >>>>>>> This looks very interesting. I'm interested in contributing. >>>>>>> >>>>>>> >>>>>>>> Thanks. >>>>>>>> -Gon >>>>>>>> >>>>>>>> --- >>>>>>>> Byung-Gon Chun >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 21, 2016 at 1:32 AM, James Malone < >>>>>>>> jamesmal...@google.com.invalid> wrote: >>>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> >>>>>>>> Attached to this message is a proposed new project - Apache >>>>>>>>> Dataflow, a >>>>>>>>> unified programming model for data processing and integration. >>>>>>>>> >>>>>>>>> The text of the proposal is included below. Additionally, the >>>>>>>>> proposal >>>>>>>>> is >>>>>>>>> in draft form on the wiki where we will make any required changes: >>>>>>>>> >>>>>>>>> https://wiki.apache.org/incubator/DataflowProposal >>>>>>>>> >>>>>>>>> We look forward to your feedback and input. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> James >>>>>>>>> >>>>>>>>> ---- >>>>>>>>> >>>>>>>>> = Apache Dataflow = >>>>>>>>> >>>>>>>>> == Abstract == >>>>>>>>> >>>>>>>>> Dataflow is an open source, unified model and set of >>>>>>>>> language-specific >>>>>>>>> SDKs >>>>>>>>> for defining and executing data processing workflows, and also data >>>>>>>>> ingestion and integration flows, supporting Enterprise Integration >>>>>>>>> Patterns >>>>>>>>> (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines >>>>>>>>> simplify >>>>>>>>> the mechanics of large-scale batch and streaming data processing >>>>>>>>> and >>>>>>>>> can >>>>>>>>> run on a number of runtimes like Apache Flink, Apache Spark, and >>>>>>>>> Google >>>>>>>>> Cloud Dataflow (a cloud service). Dataflow also brings DSL in >>>>>>>>> different >>>>>>>>> languages, allowing users to easily implement their data >>>>>>>>> integration >>>>>>>>> processes. >>>>>>>>> >>>>>>>>> == Proposal == >>>>>>>>> >>>>>>>>> Dataflow is a simple, flexible, and powerful system for distributed >>>>>>>>> data >>>>>>>>> processing at any scale. Dataflow provides a unified programming >>>>>>>>> model, a >>>>>>>>> software development kit to define and construct data processing >>>>>>>>> pipelines, >>>>>>>>> and runners to execute Dataflow pipelines in several runtime >>>>>>>>> engines, >>>>>>>>> like >>>>>>>>> Apache Spark, Apache Flink, or Google Cloud Dataflow. Dataflow can >>>>>>>>> be >>>>>>>>> used >>>>>>>>> for a variety of streaming or batch data processing goals including >>>>>>>>> ETL, >>>>>>>>> stream analysis, and aggregate computation. The underlying >>>>>>>>> programming >>>>>>>>> model for Dataflow provides MapReduce-like parallelism, combined >>>>>>>>> with >>>>>>>>> support for powerful data windowing, and fine-grained correctness >>>>>>>>> control. >>>>>>>>> >>>>>>>>> == Background == >>>>>>>>> >>>>>>>>> Dataflow started as a set of Google projects focused on making data >>>>>>>>> processing easier, faster, and less costly. The Dataflow model is a >>>>>>>>> successor to MapReduce, FlumeJava, and Millwheel inside Google and >>>>>>>>> is >>>>>>>>> focused on providing a unified solution for batch and stream >>>>>>>>> processing. >>>>>>>>> These projects on which Dataflow is based have been published in >>>>>>>>> several >>>>>>>>> papers made available to the public: >>>>>>>>> >>>>>>>>> * MapReduce - http://research.google.com/archive/mapreduce.html >>>>>>>>> >>>>>>>>> * Dataflow model - >>>>>>>>> http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf >>>>>>>>> >>>>>>>>> * FlumeJava - http://notes.stephenholiday.com/FlumeJava.pdf >>>>>>>>> >>>>>>>>> * MillWheel - http://research.google.com/pubs/pub41378.html >>>>>>>>> >>>>>>>>> Dataflow was designed from the start to provide a portable >>>>>>>>> programming >>>>>>>>> layer. When you define a data processing pipeline with the Dataflow >>>>>>>>> model, >>>>>>>>> you are creating a job which is capable of being processed by any >>>>>>>>> number >>>>>>>>> of >>>>>>>>> Dataflow processing engines. Several engines have been developed to >>>>>>>>> run >>>>>>>>> Dataflow pipelines in other open source runtimes, including a >>>>>>>>> Dataflow >>>>>>>>> runner for Apache Flink and Apache Spark. There is also a “direct >>>>>>>>> runner”, >>>>>>>>> for execution on the developer machine (mainly for dev/debug >>>>>>>>> purposes). >>>>>>>>> Another runner allows a Dataflow program to run on a managed >>>>>>>>> service, >>>>>>>>> Google Cloud Dataflow, in Google Cloud Platform. The Dataflow Java >>>>>>>>> SDK >>>>>>>>> is >>>>>>>>> already available on GitHub, and independent from the Google Cloud >>>>>>>>> Dataflow >>>>>>>>> service. Another Python SDK is currently in active development. >>>>>>>>> >>>>>>>>> In this proposal, the Dataflow SDKs, model, and a set of runners >>>>>>>>> will >>>>>>>>> be >>>>>>>>> submitted as an OSS project under the ASF. The runners which are a >>>>>>>>> part >>>>>>>>> of >>>>>>>>> this proposal include those for Spark (from Cloudera), Flink (from >>>>>>>>> data >>>>>>>>> Artisans), and local development (from Google); the Google Cloud >>>>>>>>> Dataflow >>>>>>>>> service runner is not included in this proposal. Further references >>>>>>>>> to >>>>>>>>> Dataflow will refer to the Dataflow model, SDKs, and runners which >>>>>>>>> are >>>>>>>>> a >>>>>>>>> part of this proposal (Apache Dataflow) only. The initial >>>>>>>>> submission >>>>>>>>> will >>>>>>>>> contain the already-released Java SDK; Google intends to submit the >>>>>>>>> Python >>>>>>>>> SDK later in the incubation process. The Google Cloud Dataflow >>>>>>>>> service >>>>>>>>> will >>>>>>>>> continue to be one of many runners for Dataflow, built on Google >>>>>>>>> Cloud >>>>>>>>> Platform, to run Dataflow pipelines. Necessarily, Cloud Dataflow >>>>>>>>> will >>>>>>>>> develop against the Apache project additions, updates, and changes. >>>>>>>>> Google >>>>>>>>> Cloud Dataflow will become one user of Apache Dataflow and will >>>>>>>>> participate >>>>>>>>> in the project openly and publicly. >>>>>>>>> >>>>>>>>> The Dataflow programming model has been designed with simplicity, >>>>>>>>> scalability, and speed as key tenants. In the Dataflow model, you >>>>>>>>> only >>>>>>>>> need >>>>>>>>> to think about four top-level concepts when constructing your data >>>>>>>>> processing job: >>>>>>>>> >>>>>>>>> * Pipelines - The data processing job made of a series of >>>>>>>>> computations >>>>>>>>> including input, processing, and output >>>>>>>>> >>>>>>>>> * PCollections - Bounded (or unbounded) datasets which represent >>>>>>>>> the >>>>>>>>> input, >>>>>>>>> intermediate and output data in pipelines >>>>>>>>> >>>>>>>>> * PTransforms - A data processing step in a pipeline in which one >>>>>>>>> or >>>>>>>>> more >>>>>>>>> PCollections are an input and output >>>>>>>>> >>>>>>>>> * I/O Sources and Sinks - APIs for reading and writing data which >>>>>>>>> are >>>>>>>>> the >>>>>>>>> roots and endpoints of the pipeline >>>>>>>>> >>>>>>>>> == Rationale == >>>>>>>>> >>>>>>>>> With Dataflow, Google intended to develop a framework which allowed >>>>>>>>> developers to be maximally productive in defining the processing, >>>>>>>>> and >>>>>>>>> then >>>>>>>>> be able to execute the program at various levels of >>>>>>>>> latency/cost/completeness without re-architecting or re-writing it. >>>>>>>>> This >>>>>>>>> goal was informed by Google’s past experience developing several >>>>>>>>> models, >>>>>>>>> frameworks, and tools useful for large-scale and distributed data >>>>>>>>> processing. While Google has previously published papers describing >>>>>>>>> some >>>>>>>>> of >>>>>>>>> its technologies, Google decided to take a different approach with >>>>>>>>> Dataflow. Google open-sourced the SDK and model alongside >>>>>>>>> commercialization >>>>>>>>> of the idea and ahead of publishing papers on the topic. As a >>>>>>>>> result, a >>>>>>>>> number of open source runtimes exist for Dataflow, such as the >>>>>>>>> Apache >>>>>>>>> Flink >>>>>>>>> and Apache Spark runners. >>>>>>>>> >>>>>>>>> We believe that submitting Dataflow as an Apache project will >>>>>>>>> provide >>>>>>>>> an >>>>>>>>> immediate, worthwhile, and substantial contribution to the open >>>>>>>>> source >>>>>>>>> community. As an incubating project, we believe Dataflow will have >>>>>>>>> a >>>>>>>>> better >>>>>>>>> opportunity to provide a meaningful contribution to OSS and also >>>>>>>>> integrate >>>>>>>>> with other Apache projects. >>>>>>>>> >>>>>>>>> In the long term, we believe Dataflow can be a powerful abstraction >>>>>>>>> layer >>>>>>>>> for data processing. By providing an abstraction layer for data >>>>>>>>> pipelines >>>>>>>>> and processing, data workflows can be increasingly portable, >>>>>>>>> resilient >>>>>>>>> to >>>>>>>>> breaking changes in tooling, and compatible across many execution >>>>>>>>> engines, >>>>>>>>> runtimes, and open source projects. >>>>>>>>> >>>>>>>>> == Initial Goals == >>>>>>>>> >>>>>>>>> We are breaking our initial goals into immediate (< 2 months), >>>>>>>>> short-term >>>>>>>>> (2-4 months), and intermediate-term (> 4 months). >>>>>>>>> >>>>>>>>> Our immediate goals include the following: >>>>>>>>> >>>>>>>>> * Plan for reconciling the Dataflow Java SDK and various runners >>>>>>>>> into >>>>>>>>> one >>>>>>>>> project >>>>>>>>> >>>>>>>>> * Plan for refactoring the existing Java SDK for better >>>>>>>>> extensibility >>>>>>>>> by >>>>>>>>> SDK and runner writers >>>>>>>>> >>>>>>>>> * Validating all dependencies are ASL 2.0 or compatible >>>>>>>>> >>>>>>>>> * Understanding and adapting to the Apache development process >>>>>>>>> >>>>>>>>> Our short-term goals include: >>>>>>>>> >>>>>>>>> * Moving the newly-merged lists, and build utilities to Apache >>>>>>>>> >>>>>>>>> * Start refactoring codebase and move code to Apache Git repo >>>>>>>>> >>>>>>>>> * Continue development of new features, functions, and fixes in the >>>>>>>>> Dataflow Java SDK, and Dataflow runners >>>>>>>>> >>>>>>>>> * Cleaning up the Dataflow SDK sources and crafting a roadmap and >>>>>>>>> plan >>>>>>>>> for >>>>>>>>> how to include new major ideas, modules, and runtimes >>>>>>>>> >>>>>>>>> * Establishment of easy and clear build/test framework for Dataflow >>>>>>>>> and >>>>>>>>> associated runtimes; creation of testing, rollback, and validation >>>>>>>>> policy >>>>>>>>> >>>>>>>>> * Analysis and design for work needed to make Dataflow a better >>>>>>>>> data >>>>>>>>> processing abstraction layer for multiple open source frameworks >>>>>>>>> and >>>>>>>>> environments >>>>>>>>> >>>>>>>>> Finally, we have a number of intermediate-term goals: >>>>>>>>> >>>>>>>>> * Roadmapping, planning, and execution of integrations with other >>>>>>>>> OSS >>>>>>>>> and >>>>>>>>> non-OSS projects/products >>>>>>>>> >>>>>>>>> * Inclusion of additional SDK for Python, which is under active >>>>>>>>> development >>>>>>>>> >>>>>>>>> == Current Status == >>>>>>>>> >>>>>>>>> === Meritocracy === >>>>>>>>> >>>>>>>>> Dataflow was initially developed based on ideas from many employees >>>>>>>>> within >>>>>>>>> Google. As an ASL OSS project on GitHub, the Dataflow SDK has >>>>>>>>> received >>>>>>>>> contributions from data Artisans, Cloudera Labs, and other >>>>>>>>> individual >>>>>>>>> developers. As a project under incubation, we are committed to >>>>>>>>> expanding >>>>>>>>> our effort to build an environment which supports a meritocracy. We >>>>>>>>> are >>>>>>>>> focused on engaging the community and other related projects for >>>>>>>>> support >>>>>>>>> and contributions. Moreover, we are committed to ensure >>>>>>>>> contributors >>>>>>>>> and >>>>>>>>> committers to Dataflow come from a broad mix of organizations >>>>>>>>> through a >>>>>>>>> merit-based decision process during incubation. We believe strongly >>>>>>>>> in >>>>>>>>> the >>>>>>>>> Dataflow model and are committed to growing an inclusive community >>>>>>>>> of >>>>>>>>> Dataflow contributors. >>>>>>>>> >>>>>>>>> === Community === >>>>>>>>> >>>>>>>>> The core of the Dataflow Java SDK has been developed by Google for >>>>>>>>> use >>>>>>>>> with >>>>>>>>> Google Cloud Dataflow. Google has active community engagement in >>>>>>>>> the >>>>>>>>> SDK >>>>>>>>> GitHub repository ( >>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK >>>>>>>>> ), >>>>>>>>> on Stack Overflow ( >>>>>>>>> http://stackoverflow.com/questions/tagged/google-cloud-dataflow) >>>>>>>>> and >>>>>>>>> has >>>>>>>>> had contributions from a number of organizations and indivuduals. >>>>>>>>> >>>>>>>>> Everyday, Cloud Dataflow is actively used by a number of >>>>>>>>> organizations >>>>>>>>> and >>>>>>>>> institutions for batch and stream processing of data. We believe >>>>>>>>> acceptance >>>>>>>>> will allow us to consolidate existing Dataflow-related work, grow >>>>>>>>> the >>>>>>>>> Dataflow community, and deepen connections between Dataflow and >>>>>>>>> other >>>>>>>>> open >>>>>>>>> source projects. >>>>>>>>> >>>>>>>>> === Core Developers === >>>>>>>>> >>>>>>>>> The core developers for Dataflow and the Dataflow runners are: >>>>>>>>> >>>>>>>>> * Frances Perry >>>>>>>>> >>>>>>>>> * Tyler Akidau >>>>>>>>> >>>>>>>>> * Davor Bonaci >>>>>>>>> >>>>>>>>> * Luke Cwik >>>>>>>>> >>>>>>>>> * Ben Chambers >>>>>>>>> >>>>>>>>> * Kenn Knowles >>>>>>>>> >>>>>>>>> * Dan Halperin >>>>>>>>> >>>>>>>>> * Daniel Mills >>>>>>>>> >>>>>>>>> * Mark Shields >>>>>>>>> >>>>>>>>> * Craig Chambers >>>>>>>>> >>>>>>>>> * Maximilian Michels >>>>>>>>> >>>>>>>>> * Tom White >>>>>>>>> >>>>>>>>> * Josh Wills >>>>>>>>> >>>>>>>>> === Alignment === >>>>>>>>> >>>>>>>>> The Dataflow SDK can be used to create Dataflow pipelines which can >>>>>>>>> be >>>>>>>>> executed on Apache Spark or Apache Flink. Dataflow is also related >>>>>>>>> to >>>>>>>>> other >>>>>>>>> Apache projects, such as Apache Crunch. We plan on expanding >>>>>>>>> functionality >>>>>>>>> for Dataflow runners, support for additional domain specific >>>>>>>>> languages, >>>>>>>>> and >>>>>>>>> increased portability so Dataflow is a powerful abstraction layer >>>>>>>>> for >>>>>>>>> data >>>>>>>>> processing. >>>>>>>>> >>>>>>>>> == Known Risks == >>>>>>>>> >>>>>>>>> === Orphaned Products === >>>>>>>>> >>>>>>>>> The Dataflow SDK is presently used by several organizations, from >>>>>>>>> small >>>>>>>>> startups to Fortune 100 companies, to construct production >>>>>>>>> pipelines >>>>>>>>> which >>>>>>>>> are executed in Google Cloud Dataflow. Google has a long-term >>>>>>>>> commitment >>>>>>>>> to >>>>>>>>> advance the Dataflow SDK; moreover, Dataflow is seeing increasing >>>>>>>>> interest, >>>>>>>>> development, and adoption from organizations outside of Google. >>>>>>>>> >>>>>>>>> === Inexperience with Open Source === >>>>>>>>> >>>>>>>>> Google believes strongly in open source and the exchange of >>>>>>>>> information >>>>>>>>> to >>>>>>>>> advance new ideas and work. Examples of this commitment are active >>>>>>>>> OSS >>>>>>>>> projects such as Chromium (https://www.chromium.org) and >>>>>>>>> Kubernetes >>>>>>>>> ( >>>>>>>>> http://kubernetes.io/). With Dataflow, we have tried to be >>>>>>>>> increasingly >>>>>>>>> open and forward-looking; we have published a paper in the VLDB >>>>>>>>> conference >>>>>>>>> describing the Dataflow model ( >>>>>>>>> http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to >>>>>>>>> release >>>>>>>>> the Dataflow SDK as open source software with the launch of Cloud >>>>>>>>> Dataflow. >>>>>>>>> Our submission to the Apache Software Foundation is a logical >>>>>>>>> extension >>>>>>>>> of >>>>>>>>> our commitment to open source software. >>>>>>>>> >>>>>>>>> === Homogeneous Developers === >>>>>>>>> >>>>>>>>> The majority of committers in this proposal belong to Google due to >>>>>>>>> the >>>>>>>>> fact that Dataflow has emerged from several internal Google >>>>>>>>> projects. >>>>>>>>> This >>>>>>>>> proposal also includes committers outside of Google who are >>>>>>>>> actively >>>>>>>>> involved with other Apache projects, such as Hadoop, Flink, and >>>>>>>>> Spark. >>>>>>>>> We >>>>>>>>> expect our entry into incubation will allow us to expand the number >>>>>>>>> of >>>>>>>>> individuals and organizations participating in Dataflow >>>>>>>>> development. >>>>>>>>> Additionally, separation of the Dataflow SDK from Google Cloud >>>>>>>>> Dataflow >>>>>>>>> allows us to focus on the open source SDK and model and do what is >>>>>>>>> best >>>>>>>>> for >>>>>>>>> this project. >>>>>>>>> >>>>>>>>> === Reliance on Salaried Developers === >>>>>>>>> >>>>>>>>> The Dataflow SDK and Dataflow runners have been developed primarily >>>>>>>>> by >>>>>>>>> salaried developers supporting the Google Cloud Dataflow project. >>>>>>>>> While >>>>>>>>> the >>>>>>>>> Dataflow SDK and Cloud Dataflow have been developed by different >>>>>>>>> teams >>>>>>>>> (and >>>>>>>>> this proposal would reinforce that separation) we expect our >>>>>>>>> initial >>>>>>>>> set >>>>>>>>> of >>>>>>>>> developers will still primarily be salaried. Contribution has not >>>>>>>>> been >>>>>>>>> exclusively from salaried developers, however. For example, the >>>>>>>>> contrib >>>>>>>>> directory of the Dataflow SDK ( >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib >>>>>>>>> ) >>>>>>>>> contains items from free-time contributors. Moreover, seperate >>>>>>>>> projects, >>>>>>>>> such as ScalaFlow (https://github.com/darkjh/scalaflow) have been >>>>>>>>> created >>>>>>>>> around the Dataflow model and SDK. We expect our reliance on >>>>>>>>> salaried >>>>>>>>> developers will decrease over time during incubation. >>>>>>>>> >>>>>>>>> === Relationship with other Apache products === >>>>>>>>> >>>>>>>>> Dataflow directly interoperates with or utilizes several existing >>>>>>>>> Apache >>>>>>>>> projects. >>>>>>>>> >>>>>>>>> * Build >>>>>>>>> >>>>>>>>> ** Apache Maven >>>>>>>>> >>>>>>>>> * Data I/O, Libraries >>>>>>>>> >>>>>>>>> ** Apache Avro >>>>>>>>> >>>>>>>>> ** Apache Commons >>>>>>>>> >>>>>>>>> * Dataflow runners >>>>>>>>> >>>>>>>>> ** Apache Flink >>>>>>>>> >>>>>>>>> ** Apache Spark >>>>>>>>> >>>>>>>>> Dataflow when used in batch mode shares similarities with Apache >>>>>>>>> Crunch; >>>>>>>>> however, Dataflow is focused on a model, SDK, and abstraction layer >>>>>>>>> beyond >>>>>>>>> Spark and Hadoop (MapReduce.) One key goal of Dataflow is to >>>>>>>>> provide >>>>>>>>> an >>>>>>>>> intermediate abstraction layer which can easily be implemented and >>>>>>>>> utilized >>>>>>>>> across several different processing frameworks. >>>>>>>>> >>>>>>>>> === An excessive fascination with the Apache brand === >>>>>>>>> >>>>>>>>> With this proposal we are not seeking attention or publicity. >>>>>>>>> Rather, >>>>>>>>> we >>>>>>>>> firmly believe in the Dataflow model, SDK, and the ability to make >>>>>>>>> Dataflow >>>>>>>>> a powerful yet simple framework for data processing. While the >>>>>>>>> Dataflow >>>>>>>>> SDK >>>>>>>>> and model have been open source, we believe putting code on GitHub >>>>>>>>> can >>>>>>>>> only >>>>>>>>> go so far. We see the Apache community, processes, and mission as >>>>>>>>> critical >>>>>>>>> for ensuring the Dataflow SDK and model are truly community-driven, >>>>>>>>> positively impactful, and innovative open source software. While >>>>>>>>> Google >>>>>>>>> has >>>>>>>>> taken a number of steps to advance its various open source >>>>>>>>> projects, >>>>>>>>> we >>>>>>>>> believe Dataflow is a great fit for the Apache Software Foundation >>>>>>>>> due >>>>>>>>> to >>>>>>>>> its focus on data processing and its relationships to existing ASF >>>>>>>>> projects. >>>>>>>>> >>>>>>>>> == Documentation == >>>>>>>>> >>>>>>>>> The following documentation is relevant to this proposal. Relevant >>>>>>>>> portion >>>>>>>>> of the documentation will be contributed to the Apache Dataflow >>>>>>>>> project. >>>>>>>>> >>>>>>>>> * Dataflow website: https://cloud.google.com/dataflow >>>>>>>>> >>>>>>>>> * Dataflow programming model: >>>>>>>>> https://cloud.google.com/dataflow/model/programming-model >>>>>>>>> >>>>>>>>> * Codebases >>>>>>>>> >>>>>>>>> ** Dataflow Java SDK: >>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK >>>>>>>>> >>>>>>>>> ** Flink Dataflow runner: >>>>>>>>> https://github.com/dataArtisans/flink-dataflow >>>>>>>>> >>>>>>>>> ** Spark Dataflow runner: >>>>>>>>> https://github.com/cloudera/spark-dataflow >>>>>>>>> >>>>>>>>> * Dataflow Java SDK issue tracker: >>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues >>>>>>>>> >>>>>>>>> * google-cloud-dataflow tag on Stack Overflow: >>>>>>>>> http://stackoverflow.com/questions/tagged/google-cloud-dataflow >>>>>>>>> >>>>>>>>> == Initial Source == >>>>>>>>> >>>>>>>>> The initial source for Dataflow which we will submit to the Apache >>>>>>>>> Foundation will include several related projects which are >>>>>>>>> currently >>>>>>>>> hosted >>>>>>>>> on the GitHub repositories: >>>>>>>>> >>>>>>>>> * Dataflow Java SDK ( >>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK) >>>>>>>>> >>>>>>>>> * Flink Dataflow runner ( >>>>>>>>> https://github.com/dataArtisans/flink-dataflow) >>>>>>>>> >>>>>>>>> * Spark Dataflow runner ( >>>>>>>>> https://github.com/cloudera/spark-dataflow) >>>>>>>>> >>>>>>>>> These projects have always been Apache 2.0 licensed. We intend to >>>>>>>>> bundle >>>>>>>>> all of these repositories since they are all complimentary and >>>>>>>>> should >>>>>>>>> be >>>>>>>>> maintained in one project. Prior to our submission, we will combine >>>>>>>>> all >>>>>>>>> of >>>>>>>>> these projects into a new git repository. >>>>>>>>> >>>>>>>>> == Source and Intellectual Property Submission Plan == >>>>>>>>> >>>>>>>>> The source for the Dataflow SDK and the three runners (Spark, >>>>>>>>> Flink, >>>>>>>>> Google >>>>>>>>> Cloud Dataflow) are already licensed under an Apache 2 license. >>>>>>>>> >>>>>>>>> * Dataflow SDK - >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE >>>>>>>>> >>>>>>>>> * Flink runner - >>>>>>>>> https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE >>>>>>>>> >>>>>>>>> * Spark runner - >>>>>>>>> https://github.com/cloudera/spark-dataflow/blob/master/LICENSE >>>>>>>>> >>>>>>>>> Contributors to the Dataflow SDK have also signed the Google >>>>>>>>> Individual >>>>>>>>> Contributor License Agreement ( >>>>>>>>> https://cla.developers.google.com/about/google-individual) in >>>>>>>>> order >>>>>>>>> to >>>>>>>>> contribute to the project. >>>>>>>>> >>>>>>>>> With respect to trademark rights, Google does not hold a trademark >>>>>>>>> on >>>>>>>>> the >>>>>>>>> phrase “Dataflow.” Based on feedback and guidance we receive during >>>>>>>>> the >>>>>>>>> incubation process, we are open to renaming the project if >>>>>>>>> necessary >>>>>>>>> for >>>>>>>>> trademark or other concerns. >>>>>>>>> >>>>>>>>> == External Dependencies == >>>>>>>>> >>>>>>>>> All external dependencies are licensed under an Apache 2.0 or >>>>>>>>> Apache-compatible license. As we grow the Dataflow community we >>>>>>>>> will >>>>>>>>> configure our build process to require and validate all >>>>>>>>> contributions >>>>>>>>> and >>>>>>>>> dependencies are licensed under the Apache 2.0 license or are under >>>>>>>>> an >>>>>>>>> Apache-compatible license. >>>>>>>>> >>>>>>>>> == Required Resources == >>>>>>>>> >>>>>>>>> === Mailing Lists === >>>>>>>>> >>>>>>>>> We currently use a mix of mailing lists. We will migrate our >>>>>>>>> existing >>>>>>>>> mailing lists to the following: >>>>>>>>> >>>>>>>>> * d...@dataflow.incubator.apache.org >>>>>>>>> >>>>>>>>> * u...@dataflow.incubator.apache.org >>>>>>>>> >>>>>>>>> * priv...@dataflow.incubator.apache.org >>>>>>>>> >>>>>>>>> * comm...@dataflow.incubator.apache.org >>>>>>>>> >>>>>>>>> === Source Control === >>>>>>>>> >>>>>>>>> The Dataflow team currently uses Git and would like to continue to >>>>>>>>> do >>>>>>>>> so. >>>>>>>>> We request a Git repository for Dataflow with mirroring to GitHub >>>>>>>>> enabled. >>>>>>>>> >>>>>>>>> === Issue Tracking === >>>>>>>>> >>>>>>>>> We request the creation of an Apache-hosted JIRA. The Dataflow >>>>>>>>> project >>>>>>>>> is >>>>>>>>> currently using both a public GitHub issue tracker and internal >>>>>>>>> Google >>>>>>>>> issue tracking. We will migrate and combine from these two sources >>>>>>>>> to >>>>>>>>> the >>>>>>>>> Apache JIRA. >>>>>>>>> >>>>>>>>> == Initial Committers == >>>>>>>>> >>>>>>>>> * Aljoscha Krettek [aljos...@apache.org] >>>>>>>>> >>>>>>>>> * Amit Sela [amitsel...@gmail.com] >>>>>>>>> >>>>>>>>> * Ben Chambers [bchamb...@google.com] >>>>>>>>> >>>>>>>>> * Craig Chambers [chamb...@google.com] >>>>>>>>> >>>>>>>>> * Dan Halperin [dhalp...@google.com] >>>>>>>>> >>>>>>>>> * Davor Bonaci [da...@google.com] >>>>>>>>> >>>>>>>>> * Frances Perry [f...@google.com] >>>>>>>>> >>>>>>>>> * James Malone [jamesmal...@google.com] >>>>>>>>> >>>>>>>>> * Jean-Baptiste Onofré [jbono...@apache.org] >>>>>>>>> >>>>>>>>> * Josh Wills [jwi...@apache.org] >>>>>>>>> >>>>>>>>> * Kostas Tzoumas [kos...@data-artisans.com] >>>>>>>>> >>>>>>>>> * Kenneth Knowles [k...@google.com] >>>>>>>>> >>>>>>>>> * Luke Cwik [lc...@google.com] >>>>>>>>> >>>>>>>>> * Maximilian Michels [m...@apache.org] >>>>>>>>> >>>>>>>>> * Stephan Ewen [step...@data-artisans.com] >>>>>>>>> >>>>>>>>> * Tom White [t...@cloudera.com] >>>>>>>>> >>>>>>>>> * Tyler Akidau [taki...@google.com] >>>>>>>>> >>>>>>>>> == Affiliations == >>>>>>>>> >>>>>>>>> The initial committers are from six organizations. Google developed >>>>>>>>> Dataflow and the Dataflow SDK, data Artisans developed the Flink >>>>>>>>> runner, >>>>>>>>> and Cloudera (Labs) developed the Spark runner. >>>>>>>>> >>>>>>>>> * Cloudera >>>>>>>>> >>>>>>>>> ** Tom White >>>>>>>>> >>>>>>>>> * Data Artisans >>>>>>>>> >>>>>>>>> ** Aljoscha Krettek >>>>>>>>> >>>>>>>>> ** Kostas Tzoumas >>>>>>>>> >>>>>>>>> ** Maximilian Michels >>>>>>>>> >>>>>>>>> ** Stephan Ewen >>>>>>>>> >>>>>>>>> * Google >>>>>>>>> >>>>>>>>> ** Ben Chambers >>>>>>>>> >>>>>>>>> ** Dan Halperin >>>>>>>>> >>>>>>>>> ** Davor Bonaci >>>>>>>>> >>>>>>>>> ** Frances Perry >>>>>>>>> >>>>>>>>> ** James Malone >>>>>>>>> >>>>>>>>> ** Kenneth Knowles >>>>>>>>> >>>>>>>>> ** Luke Cwik >>>>>>>>> >>>>>>>>> ** Tyler Akidau >>>>>>>>> >>>>>>>>> * PayPal >>>>>>>>> >>>>>>>>> ** Amit Sela >>>>>>>>> >>>>>>>>> * Slack >>>>>>>>> >>>>>>>>> ** Josh Wills >>>>>>>>> >>>>>>>>> * Talend >>>>>>>>> >>>>>>>>> ** Jean-Baptiste Onofré >>>>>>>>> >>>>>>>>> == Sponsors == >>>>>>>>> >>>>>>>>> === Champion === >>>>>>>>> >>>>>>>>> * Jean-Baptiste Onofre [jbono...@apache.org] >>>>>>>>> >>>>>>>>> === Nominated Mentors === >>>>>>>>> >>>>>>>>> * Jim Jagielski [j...@apache.org] >>>>>>>>> >>>>>>>>> * Venkatesh Seetharam [venkat...@apache.org] >>>>>>>>> >>>>>>>>> * Bertrand Delacretaz [bdelacre...@apache.org] >>>>>>>>> >>>>>>>>> * Ted Dunning [tdunn...@apache.org] >>>>>>>>> >>>>>>>>> === Sponsoring Entity === >>>>>>>>> >>>>>>>>> The Apache Incubator >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Jean-Baptiste Onofré >>>>>>> jbono...@apache.org >>>>>>> http://blog.nanthrax.net >>>>>>> Talend - http://www.talend.com >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>>>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>> >>>>> Jean-Baptiste Onofré >>>>> jbono...@apache.org >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>>> >>>>> >>>>> >>>>> >>>> -- >>> Jean-Baptiste Onofré >>> jbono...@apache.org >>> http://blog.nanthrax.net >>> Talend - http://www.talend.com >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>> For additional commands, e-mail: general-h...@incubator.apache.org >>> >>> >>> >> > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- Thanks and Regards, Mayank Cell: 408-718-9370
