Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

Thank you for your work in the community and Congratulations!! :) On Thu, May 2, 2019 at 9:44 PM Ankur Goenka wrote: > Congratulations and thank you for making Beam awesome! > > *From: *Chamikara Jayalath > *Date: *Thu, May 2, 2019, 4:03 PM > *To: *dev > > Congratulations! >> >> On Thu, May 2,

Re: PardoLifeCycle: Teardown after failed call to setup

On Mon, May 6, 2019 at 2:19 PM Reuven Lax wrote: > > > On Mon, May 6, 2019 at 2:06 PM Kenneth Knowles wrote: > >> The specification of TearDown is that it is best effort, certainly. >> > > Though I believe the intent of that specification was that a runner will > call it as long as the process i

Re: [DISCUSS] Reconciling ValueState in Java and Python (was: [docs] Python State & Timers)

When used as metadata I think the ReadModifyWrite naming is very accurate for the majority of cases. The only case that does not follow that pattern is if its being used as a Boolean to indicate that something should be done in the OnTimer call based on an event that has been seen in the OnProcess

Re: [Discuss] Publishing pre-release artifacts to repositories

Thanks Ahmet for the time you put into this. AFAIU Roberts proposal resolves my concerns. On Mon, May 6, 2019 at 8:01 PM Ahmet Altay wrote: > Thank you Max. Michael, does the current state of the proposal address > your concerns? > > *From: *Maximilian Michels > *Date: *Mon, May 6, 2019 at 8:32

Re: Beam at Google Summer of Code 2019

Welcome, Tanay! Nice proposal. *From: *Kenneth Knowles *Date: *Mon, May 6, 2019 at 4:13 PM *To: *dev *Cc: * Nice! Welcome, Tanay! > > On Mon, May 6, 2019 at 3:56 PM Reza Rokni wrote: > >> +1 Congratulations! Looking forward to using your work ! >> >> *From: *Valentyn Tymofieiev >> *Date: *Tue

Re: Beam at Google Summer of Code 2019

Nice! Welcome, Tanay! On Mon, May 6, 2019 at 3:56 PM Reza Rokni wrote: > +1 Congratulations! Looking forward to using your work ! > > *From: *Valentyn Tymofieiev > *Date: *Tue, 7 May 2019 at 05:40 > *To: *dev > *Cc: * > > Congrats & good luck Tanay & Pablo! >> >> *From: *Connell O'Callaghan >

Re: Beam at Google Summer of Code 2019

+1 Congratulations! Looking forward to using your work ! *From: *Valentyn Tymofieiev *Date: *Tue, 7 May 2019 at 05:40 *To: *dev *Cc: * Congrats & good luck Tanay & Pablo! > > *From: *Connell O'Callaghan > *Date: *Mon, May 6, 2019 at 4:15 PM > *To: * > *Cc: * > > Well done Tanay - good luck w

Re: Better naming for runner specific options

On Mon, May 6, 2019 at 3:01 PM Ahmet Altay wrote: > There is RunnerOptions already. Its options are populated by querying the > job service. Any portable runner is able to provide a list of options that > is runner specific through that mechanism. > > *From: *Reza Rokni > *Date: *Mon, May 6, 201

Re: Better naming for runner specific options

So the options here would be moved to runner options? https://beam.apache.org/releases/pydoc/2.12.0/_modules/apache_beam/options/pipeline_options.html#WorkerOptions In Java they are in DataflowPipelineWorkerPoolOptions and of course we have FlinkPipelineOptions etc... *From: *Chamikara Jayalath

Re: Better naming for runner specific options

There is RunnerOptions already. Its options are populated by querying the job service. Any portable runner is able to provide a list of options that is runner specific through that mechanism. *From: *Reza Rokni *Date: *Mon, May 6, 2019 at 2:57 PM *To: * So the options here would be moved to run

Re: Beam at Google Summer of Code 2019

Congrats & good luck Tanay & Pablo! *From: *Connell O'Callaghan *Date: *Mon, May 6, 2019 at 4:15 PM *To: * *Cc: * Well done Tanay - good luck with this project!!! > > +1 Pablo - thank you for this mentorship!!! > > On Mon, May 6, 2019 at 1:11 PM Chamikara Jayalath > wrote: > >> Congrats Tanay

Re: Contributing to Beam

Welcome. On Mon, May 6, 2019 at 2:23 PM Reuven Lax wrote: > Welcome! > > On Mon, May 6, 2019 at 2:15 PM Kenneth Knowles wrote: > >> Welcome! >> >> On Mon, May 6, 2019 at 9:20 AM Ahmet Altay wrote: >> >>> Welcome Shehzaad! >>> >>> *From: *Ismaël Mejía >>> *Date: *Sat, May 4, 2019 at 2:30 PM >>

Re: Better naming for runner specific options

On Mon, May 6, 2019 at 2:13 PM Lukasz Cwik wrote: > There were also discussions[1] in the past about scoping PipelineOptions > to specific PTransforms. Would scoping PipelineOptions to PTransforms make > this a more general solution? > > 1: > https://lists.apache.org/thread.html/05f849d39788cb0af

Re: Contributing to Beam

Welcome! On Mon, May 6, 2019 at 2:15 PM Kenneth Knowles wrote: > Welcome! > > On Mon, May 6, 2019 at 9:20 AM Ahmet Altay wrote: > >> Welcome Shehzaad! >> >> *From: *Ismaël Mejía >> *Date: *Sat, May 4, 2019 at 2:30 PM >> *To: * >> >> Done, welcome! >>> >>> On Sat, May 4, 2019 at 3:43 AM Shehza

Re: PardoLifeCycle: Teardown after failed call to setup

On Mon, May 6, 2019 at 2:06 PM Kenneth Knowles wrote: > The specification of TearDown is that it is best effort, certainly. > Though I believe the intent of that specification was that a runner will call it as long as the process itself has not crashed. > If your runner supports it, then the t

Re: Better naming for runner specific options

There were also discussions[1] in the past about scoping PipelineOptions to specific PTransforms. Would scoping PipelineOptions to PTransforms make this a more general solution? 1: https://lists.apache.org/thread.html/05f849d39788cb0af840cb9e86ca631586783947eb4e5a1774b647d1@%3Cdev.beam.apache.org%

Re: Access to JIRA

Welcome! On Mon, May 6, 2019 at 9:28 AM Ahmet Altay wrote: > Welcome Frederik! > > *From: *Ismaël Mejía > *Date: *Sat, May 4, 2019 at 2:31 PM > *To: * > > It should work now, enjoy! >> On Sat, May 4, 2019 at 7:58 PM Frederik Bode >> wrote: >> >>> >>> Hello everyone! >>> >>> I'm working on por

Re: Contributing to Beam

Welcome! On Mon, May 6, 2019 at 9:20 AM Ahmet Altay wrote: > Welcome Shehzaad! > > *From: *Ismaël Mejía > *Date: *Sat, May 4, 2019 at 2:30 PM > *To: * > > Done, welcome! >> >> On Sat, May 4, 2019 at 3:43 AM Shehzaad Nakhoda >> wrote: >> >>> Hello >>> >>> I’m hoping to work with Rueven Lax (Go

Re: PardoLifeCycle: Teardown after failed call to setup

The specification of TearDown is that it is best effort, certainly. If your runner supports it, then the test is good to make sure there is not a regression. If your runner has partial support, that is within spec. But the idea of the spec is more than you might have such a failure that it is impos

Re: Beam at Google Summer of Code 2019

Well done Tanay - good luck with this project!!! +1 Pablo - thank you for this mentorship!!! On Mon, May 6, 2019 at 1:11 PM Chamikara Jayalath wrote: > Congrats Tanay and thanks Pablo for mentoring :) > > On Mon, May 6, 2019 at 11:53 AM Aizhamal Nurmamat kyzy < > aizha...@google.com> wrote: > >

Re: Beam at Google Summer of Code 2019

Congrats Tanay and thanks Pablo for mentoring :) On Mon, May 6, 2019 at 11:53 AM Aizhamal Nurmamat kyzy wrote: > Congratulations Tanay! > > *From: *Pablo Estrada > *Date: *Mon, May 6, 2019 at 11:34 AM > *To: *dev, > > Hello all, >> it is my pleasure to share with everyone that Tanay Tummalapal

Re: Better naming for runner specific options

Having namespaces for option makes sense. I think, along with a help command to print all the options given the runner name will be useful. As for the scope of name spacing, I think that assigning a logical name space gives more flexibility around how and where we declare options. It also make futu

Re: Beam at Google Summer of Code 2019

Congratulations Tanay! *From: *Pablo Estrada *Date: *Mon, May 6, 2019 at 11:34 AM *To: *dev, Hello all, > it is my pleasure to share with everyone that Tanay Tummalapalli has been > accepted as a GSoC student with Beam, to implement support for File Loads > into BigQuery for streaming pipelines

Beam at Google Summer of Code 2019

Hello all, it is my pleasure to share with everyone that Tanay Tummalapalli has been accepted as a GSoC student with Beam, to implement support for File Loads into BigQuery for streaming pipelines[1]. Tanay wrote a very strong proposal, and showed understanding of the tricky streaming consideratio

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

Gentle reminder. We will be cutting the 2.13.0 branch day after tomorrow ( May 8th ). Please mark the release blockers as fixed and Fix Version as 2.13.0. Thanks, Ankur On Fri, Apr 26, 2019 at 2:28 PM Ankur Goenka wrote: > Link in the download link page will be useful. > > Additionally, to not

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Thanks everyone! On Mon, May 6, 2019, 07:40 Maximilian Michels wrote: > Congrats and welcome to the committers-club! > > On 06.05.19 14:39, Gleb Kanterov wrote: > > Congratulations! > > > > On Mon, May 6, 2019 at 2:34 PM Valentyn Tymofieiev > > wrote: > > > > Con

Re: [Discuss] Publishing pre-release artifacts to repositories

Thank you Max. Michael, does the current state of the proposal address your concerns? *From: *Maximilian Michels *Date: *Mon, May 6, 2019 at 8:32 AM *To: * Thanks for the link to the distribution guideline thread. The proposed > process sounds good to me. > > Concerning Michael's point, I think

Re: [DISCUSS][SQL] Providing support for DISTINCT aggregations

Good point to reject DISTINCT operations currently, as it's not handled now. There could be more similar cases need to revise and document well. Regarding to how to DISTINCT support, I was confused by stateful CombineFn at first. To make it simple, we can extend step by step, like reject DISTINCT+

Re: Access to JIRA

Welcome Frederik! *From: *Ismaël Mejía *Date: *Sat, May 4, 2019 at 2:31 PM *To: * It should work now, enjoy! > On Sat, May 4, 2019 at 7:58 PM Frederik Bode wrote: > >> >> Hello everyone! >> >> I'm working on porting beam to python 3. Could I please get access >> to JIRA? My JIRA username is fr

Re: Contributing to Beam

Welcome Shehzaad! *From: *Ismaël Mejía *Date: *Sat, May 4, 2019 at 2:30 PM *To: * Done, welcome! > > On Sat, May 4, 2019 at 3:43 AM Shehzaad Nakhoda > wrote: > >> Hello >> >> I’m hoping to work with Rueven Lax (Google) on some enhancements and >> existing issues. >> >> I would appreciate the a

Re: Beam Dependency Check Report (2019-05-06)

The JIRA credentials are missing on the Jenkins. I'm on it. *From: *Apache Jenkins Server *Date: *Mon, May 6, 2019 at 5:06 AM *To: * ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' > does not exist

Re: [Discuss] Publishing pre-release artifacts to repositories

Thanks for the link to the distribution guideline thread. The proposed process sounds good to me. Concerning Michael's point, I think we should follow the same procedure as we do for releasing to Maven Central, i.e. create a staging repository for validation which then gets released. The eas

Re: Better naming for runner specific options

Good points. As already mentioned there is no namespacing between the different pipeline option classes. In particular, there is no separate namespace for system and user options which is most concerning. I'm in favor of an optional namespace using the class name of the defining pipeline optio

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congrats and welcome to the committers-club! On 06.05.19 14:39, Gleb Kanterov wrote: Congratulations! On Mon, May 6, 2019 at 2:34 PM Valentyn Tymofieiev > wrote: Congrats, Udi! *From: *Thomas Weise mailto:t...@apache.org>> *Date: *Mon, May 6, 2019 at 7

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congratulations Udi! On Mon, May 6, 2019 at 2:49 PM Gleb Kanterov wrote: > Congratulations! > > On Mon, May 6, 2019 at 2:34 PM Valentyn Tymofieiev > wrote: > >> Congrats, Udi! >> >> *From: *Thomas Weise >> *Date: *Mon, May 6, 2019 at 7:50 AM >> *To: * >> >> Congrats! >>> >>> >>> On Mon, May 6

PardoLifeCycle: Teardown after failed call to setup

Hi, after stumbling upon [1] and trying to implement a fix [2], ParDoLifeCycleTest are failing for direct runner, spark validatesRunnerBatch and flink validatesRunnerBatch fail as DoFns teardown is not invoked, if DoFns setup throw an exceptions. This seems to be in line with the specification [3

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congratulations! On Mon, May 6, 2019 at 2:34 PM Valentyn Tymofieiev wrote: > Congrats, Udi! > > *From: *Thomas Weise > *Date: *Mon, May 6, 2019 at 7:50 AM > *To: * > > Congrats! >> >> >> On Mon, May 6, 2019 at 2:25 AM Łukasz Gajowy wrote: >> >>> Congrats! :) >>> >>> pon., 6 maj 2019 o 10:45 A

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congrats, Udi! *From: *Thomas Weise *Date: *Mon, May 6, 2019 at 7:50 AM *To: * Congrats! > > > On Mon, May 6, 2019 at 2:25 AM Łukasz Gajowy wrote: > >> Congrats! :) >> >> pon., 6 maj 2019 o 10:45 Alexey Romanenko >> napisał(a): >> >>> Congratulations, Udi! Thanks for your work! >>> >>> On 4 M

Beam Dependency Check Report (2019-05-06)

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congrats! On Mon, May 6, 2019 at 2:25 AM Łukasz Gajowy wrote: > Congrats! :) > > pon., 6 maj 2019 o 10:45 Alexey Romanenko > napisał(a): > >> Congratulations, Udi! Thanks for your work! >> >> On 4 May 2019, at 01:24, Heejong Lee wrote: >> >> Congratulations! >> >> On Fri, May 3, 2019 at 3:53

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congrats! :) pon., 6 maj 2019 o 10:45 Alexey Romanenko napisał(a): > Congratulations, Udi! Thanks for your work! > > On 4 May 2019, at 01:24, Heejong Lee wrote: > > Congratulations! > > On Fri, May 3, 2019 at 3:53 PM Reza Rokni wrote: > >> Congratulations ! >> >> *From: *Reuven Lax >> *Date:

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congratulations, Udi! Thanks for your work! > On 4 May 2019, at 01:24, Heejong Lee wrote: > > Congratulations! > > On Fri, May 3, 2019 at 3:53 PM Reza Rokni > wrote: > Congratulations ! > > From: Reuven Lax mailto:re...@google.com>> > Date: Sat, 4 May 2019, 06:42 > To: