Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

IP Clearance has passed, so I'm just retesting the PR and merging. Kenn On Thu, Oct 4, 2018 at 3:51 PM Kenneth Knowles wrote: > I've filed the IP clearance record: > http://incubator.apache.org/ip-clearance/beam-dataflow-java-worker.html > > https://lists.apache.org/thread.html/1cc32072bd888f6b

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

I've filed the IP clearance record: http://incubator.apache.org/ip-clearance/beam-dataflow-java-worker.html https://lists.apache.org/thread.html/1cc32072bd888f6b1335f29db2cc4194ab0c70e35552c327c40122e1@%3Cgeneral.incubator.apache.org%3E Kenn On Wed, Oct 3, 2018 at 4:19 PM Boyuan Zhang wrote: >

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

Hey all, We are tracking the dataflow worker donating process here: https://issues.apache.org/jira/browse/BEAM-5634 . Boyuan Zhang On Mon, Sep 17, 2018 at 5:05 PM Lukasz Cwik wrote: > Thanks all, closing the vote with 18 +1s, 5 of which are binding. > > I'll try to get this code out and hopefu

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

Thanks all, closing the vote with 18 +1s, 5 of which are binding. I'll try to get this code out and hopefully don't have any legal issues within Google or with ASF to perform the donation. Will keep the community up to date. On Mon, Sep 17, 2018 at 3:28 PM Ankur Chauhan wrote: > +1 > > Sent fro

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 (binding) On Mon, Sep 17, 2018 at 3:27 PM Ankur Goenka wrote: > +1 > > On Sun, Sep 16, 2018 at 3:20 AM Maximilian Michels wrote: > >> +1 (binding) >> >> On 15.09.18 20:07, Reuven Lax wrote: >> > +1 >> > >> > On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > > > wrote: >> >

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 Sent from my iPhone > On Sep 17, 2018, at 15:26, Ankur Goenka wrote: > > +1 > >> On Sun, Sep 16, 2018 at 3:20 AM Maximilian Michels wrote: >> +1 (binding) >> >> On 15.09.18 20:07, Reuven Lax wrote: >> > +1 >> > >> > On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > > >

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Sun, Sep 16, 2018 at 3:20 AM Maximilian Michels wrote: > +1 (binding) > > On 15.09.18 20:07, Reuven Lax wrote: > > +1 > > > > On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > > wrote: > > > > +1 > > > > -Rui > > > > On Sat, Sep 15, 2018 at 12:32 AM Robert B

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 (binding) On 15.09.18 20:07, Reuven Lax wrote: +1 On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > wrote: +1 -Rui On Sat, Sep 15, 2018 at 12:32 AM Robert Bradshaw mailto:rober...@google.com>> wrote: +1 (binding) On Sat, Sep 15, 2018 a

Re: Donating the Dataflow Worker code to Apache Beam

If anything, merging the Dataflow Worker code shows Google's commitment to the Beam project. Yes, it does solve internal issues with syncing their runtime with Beam, but Beam was always about the programming model for data processing, not about a specific type of execution engine. Like any oth

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Sat, Sep 15, 2018 at 9:40 AM Rui Wang wrote: > +1 > > -Rui > > On Sat, Sep 15, 2018 at 12:32 AM Robert Bradshaw > wrote: > >> +1 (binding) >> >> On Sat, Sep 15, 2018 at 6:44 AM Tim wrote: >> >>> +1 >>> >>> On 15 Sep 2018, at 01:23, Yifan Zou wrote: >>> >>> +1 >>> >>> On Fri, Sep 14, 201

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 -Rui On Sat, Sep 15, 2018 at 12:32 AM Robert Bradshaw wrote: > +1 (binding) > > On Sat, Sep 15, 2018 at 6:44 AM Tim wrote: > >> +1 >> >> On 15 Sep 2018, at 01:23, Yifan Zou wrote: >> >> +1 >> >> On Fri, Sep 14, 2018 at 4:20 PM David Morávek >> wrote: >> >>> +1 >>> >>> >>> >>> On 15 Sep 20

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 (binding) On Sat, Sep 15, 2018 at 6:44 AM Tim wrote: > +1 > > On 15 Sep 2018, at 01:23, Yifan Zou wrote: > > +1 > > On Fri, Sep 14, 2018 at 4:20 PM David Morávek > wrote: > >> +1 >> >> >> >> On 15 Sep 2018, at 00:59, Anton Kedin wrote: >> >> +1 >> >> On Fri, Sep 14, 2018 at 3:22 PM Alan My

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 > On 15 Sep 2018, at 01:23, Yifan Zou wrote: > > +1 > >> On Fri, Sep 14, 2018 at 4:20 PM David Morávek >> wrote: >> +1 >> >> >> >>> On 15 Sep 2018, at 00:59, Anton Kedin wrote: >>> >>> +1 >>> On Fri, Sep 14, 2018 at 3:22 PM Alan Myrvold wrote: +1 > On Fri, Sep 14

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 4:20 PM David Morávek wrote: > +1 > > > > On 15 Sep 2018, at 00:59, Anton Kedin wrote: > > +1 > > On Fri, Sep 14, 2018 at 3:22 PM Alan Myrvold wrote: > >> +1 >> >> On Fri, Sep 14, 2018 at 3:16 PM Boyuan Zhang wrote: >> >>> +1 >>> >>> On Fri, Sep 14, 2018 at 3:15 PM

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 > On 15 Sep 2018, at 00:59, Anton Kedin wrote: > > +1 > >> On Fri, Sep 14, 2018 at 3:22 PM Alan Myrvold wrote: >> +1 >> >>> On Fri, Sep 14, 2018 at 3:16 PM Boyuan Zhang wrote: >>> +1 >>> On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde wrote: +1 > On Fri, Sep 14, 2018

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 3:22 PM Alan Myrvold wrote: > +1 > > On Fri, Sep 14, 2018 at 3:16 PM Boyuan Zhang wrote: > >> +1 >> >> On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde wrote: >> >>> +1 >>> >>> On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay wrote: >>> +1 (binding) On Fri, S

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 3:16 PM Boyuan Zhang wrote: > +1 > > On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde wrote: > >> +1 >> >> On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay wrote: >> >>> +1 (binding) >>> >>> On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik wrote: >>> +1 (binding)

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde wrote: > +1 > > On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay wrote: > >> +1 (binding) >> >> On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik wrote: >> >>> +1 (binding) >>> >>> On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada >>> wrote: >>> +1

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay wrote: > +1 (binding) > > On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik wrote: > >> +1 (binding) >> >> On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada wrote: >> >>> +1 >>> >>> On Fri, Sep 14, 2018 at 2:32 PM Andrew Pilloud >>> wrote: >>> +1 >>>

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 (binding) On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik wrote: > +1 (binding) > > On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada wrote: > >> +1 >> >> On Fri, Sep 14, 2018 at 2:32 PM Andrew Pilloud >> wrote: >> >>> +1 >>> >>> On Fri, Sep 14, 2018 at 2:31 PM Lukasz Cwik wrote: >>> There w

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 (binding) On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada wrote: > +1 > > On Fri, Sep 14, 2018 at 2:32 PM Andrew Pilloud > wrote: > >> +1 >> >> On Fri, Sep 14, 2018 at 2:31 PM Lukasz Cwik wrote: >> >>> There was generally positive support and good feedback[1] but it was not >>> unanimous. I w

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 2:32 PM Andrew Pilloud wrote: > +1 > > On Fri, Sep 14, 2018 at 2:31 PM Lukasz Cwik wrote: > >> There was generally positive support and good feedback[1] but it was not >> unanimous. I wanted to bring the donation of the Dataflow worker code base >> to Apache Beam mast

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 2:31 PM Lukasz Cwik wrote: > There was generally positive support and good feedback[1] but it was not > unanimous. I wanted to bring the donation of the Dataflow worker code base > to Apache Beam master to a vote. > > +1: Support having the Dataflow worker code as part

[VOTE] Donating the Dataflow Worker code to Apache Beam

There was generally positive support and good feedback[1] but it was not unanimous. I wanted to bring the donation of the Dataflow worker code base to Apache Beam master to a vote. +1: Support having the Dataflow worker code as part of Apache Beam master branch -1: Dataflow worker code should live

Re: Donating the Dataflow Worker code to Apache Beam

On Fri, Sep 14, 2018 at 10:02 AM Romain Manni-Bucau wrote: > > Le ven. 14 sept. 2018 à 09:48, Robert Bradshaw a > écrit : > >> On Fri, Sep 14, 2018 at 8:00 AM Romain Manni-Bucau >> wrote: >> >>> Well IBM runner is outside Beam for instance so this is not really a >>> point IMHO. >>> >>> My view

Re: Donating the Dataflow Worker code to Apache Beam

Le ven. 14 sept. 2018 à 09:48, Robert Bradshaw a écrit : > On Fri, Sep 14, 2018 at 8:00 AM Romain Manni-Bucau > wrote: > >> Well IBM runner is outside Beam for instance so this is not really a >> point IMHO. >> >> My view is simple: >> 1. does this module bring anything to Beam as a project: I u

Re: Donating the Dataflow Worker code to Apache Beam

+1 (non googler) I think this is actually a nice move. Even if there is no immediate end-user benefit (no one can directly run it), it will probably be good and valuable code for other runners to learn and borrow from, so there is benefit for other developers. Plus, it eases the life of some othe

Re: Donating the Dataflow Worker code to Apache Beam

On Fri, Sep 14, 2018 at 8:00 AM Romain Manni-Bucau wrote: > Well IBM runner is outside Beam for instance so this is not really a point > IMHO. > > My view is simple: > 1. does this module bring anything to Beam as a project: I understand your > answer as a no (please clarify if I'm wrong) > As h

Re: Donating the Dataflow Worker code to Apache Beam

Well IBM runner is outside Beam for instance so this is not really a point IMHO. My view is simple: 1. does this module bring anything to Beam as a project: I understand your answer as a no (please clarify if I'm wrong) 2. does this module bring anything to Beam or Big Data users: same answer So

Re: Donating the Dataflow Worker code to Apache Beam

Dataflow tests are part of Beam post submit, and if a PR breaks the Dataflow runner it will probably be rolled back. Today Beam contributors that make changes impacting the runner boundary have no way to make those changes without breaking Dataflow (unless they as a Googler to help them). Fortunate

Re: Donating the Dataflow Worker code to Apache Beam

Flink, Spark, Apex are usable since they are OS so you grab them+beam and you "run". If I grab dataflow worker + X OS project and "run" it is the same, however if I grab dataflow worker and cant do anything with it, the added value for Beam and users is pretty null, no? Just means Google should fin

Re: Donating the Dataflow Worker code to Apache Beam

Romain, the code is very similar to the adaptation layer between the shared libraries part of Apache Beam and any other runner, for example the code within runners/spark or runners/apex or runners/flink. If someone wanted to build an emulator of the Dataflow service, they would be able to re-use th

Re: Donating the Dataflow Worker code to Apache Beam

On Thu, Sep 13, 2018 at 12:53 PM Romain Manni-Bucau wrote: > If usable by itself without google karma (can you use a worker without > dataflow itself?) it sounds awesome otherwise it sounds weird IMHO. > Can you elaborate a bit more on using worker without dataflow? I essentially see that as o

Re: Donating the Dataflow Worker code to Apache Beam

Big +1 (non googler) Great help for transparency, future runners, learning, etc... On Thu, Sep 13, 2018 at 4:08 PM Andrew Pilloud wrote: > +1 > > On Thu, Sep 13, 2018 at 12:53 PM Romain Manni-Bucau > wrote: > >> If usable by itself without google karma (can you use a worker without >> dataflow

Re: Donating the Dataflow Worker code to Apache Beam

+1 On Thu, Sep 13, 2018 at 12:53 PM Romain Manni-Bucau wrote: > If usable by itself without google karma (can you use a worker without > dataflow itself?) it sounds awesome otherwise it sounds weird IMHO. > > Le jeu. 13 sept. 2018 21:36, Kai Jiang a écrit : > >> +1 (non googler) >> >> big help

Re: Donating the Dataflow Worker code to Apache Beam

If usable by itself without google karma (can you use a worker without dataflow itself?) it sounds awesome otherwise it sounds weird IMHO. Le jeu. 13 sept. 2018 21:36, Kai Jiang a écrit : > +1 (non googler) > > big help for transparency and for future runners. > > Best, > Kai > > On Thu, Sep 13,

Re: Donating the Dataflow Worker code to Apache Beam

+1 (non googler) big help for transparency and for future runners. Best, Kai On Thu, Sep 13, 2018, 11:45 Xinyu Liu wrote: > Big +1 (non-googler). > > From Samza Runner's perspective, we are very happy to see dataflow worker > code so we can learn and compete :). > > Thanks, > Xinyu > > On Thu,

Re: Donating the Dataflow Worker code to Apache Beam

Big +1 (non-googler). >From Samza Runner's perspective, we are very happy to see dataflow worker code so we can learn and compete :). Thanks, Xinyu On Thu, Sep 13, 2018 at 11:34 AM Suneel Marthi wrote: > +1 (non-googler) > > This is a great 👍 move > > Sent from my iPhone > > On Sep 13, 2018, a

Re: Donating the Dataflow Worker code to Apache Beam

+1 (non-googler) This is a great 👍 move Sent from my iPhone > On Sep 13, 2018, at 2:25 PM, Tim Robertson wrote: > > +1 (non googler) > It sounds pragmatic, helps with transparency should issues arise and enables > more people to fix. > > >> On Thu, Sep 13, 2018 at 8:15 PM Dan Halperin wr

Re: Donating the Dataflow Worker code to Apache Beam

+1 And I think more unit tests is a nice thing than a downside :-) -Rui On Thu, Sep 13, 2018 at 11:25 AM Tim Robertson wrote: > +1 (non googler) > It sounds pragmatic, helps with transparency should issues arise and > enables more people to fix. > > > On Thu, Sep 13, 2018 at 8:15 PM Dan Halper

Re: Donating the Dataflow Worker code to Apache Beam

+1 (non googler) It sounds pragmatic, helps with transparency should issues arise and enables more people to fix. On Thu, Sep 13, 2018 at 8:15 PM Dan Halperin wrote: > From my perspective as a (non-Google) community member, huge +1. > > I don't see anything bad for the community about open sour

Re: Donating the Dataflow Worker code to Apache Beam

>From my perspective as a (non-Google) community member, huge +1. I don't see anything bad for the community about open sourcing more of the probably-most-used runner. While the DirectRunner is probably still the most referential implementation of Beam, can't hurt to see more working code. Other r

Re: Donating the Dataflow Worker code to Apache Beam

Yes, I'm specifically asking the community for opinions as to whether it should be accepted or not. On Thu, Sep 13, 2018 at 10:51 AM Raghu Angadi wrote: > This is terrific! > > Is thread asking for opinions from the community about if it should be > accepted? Assuming Google side decision is mad

Re: Donating the Dataflow Worker code to Apache Beam

This is terrific! Is thread asking for opinions from the community about if it should be accepted? Assuming Google side decision is made to contribute, big +1 from me to include it next to other runners. On Thu, Sep 13, 2018 at 10:38 AM Lukasz Cwik wrote: > At Google we have been importing the

Re: Donating the Dataflow Worker code to Apache Beam

There have been multiple scenarios where people changed Beam, and ended up breaking the Dataflow runner because that code lived in a private repository. I believe that putting the Dataflow runner code in the public repository will make it easier and simpler to make changes to Apache Beam. Reuven

Donating the Dataflow Worker code to Apache Beam

At Google we have been importing the Apache Beam code base and integrating it with the Google portion of the codebase that supports the Dataflow worker. This process is painful as we regularly are making breaking API changes to support libraries related to running portable pipelines (and sometimes