Re: Parallelizing test runs

2018-07-03 Thread Rafael Fernandez
Summary for all folks following this story -- and many thanks for explaining configs to me and pointing me to files and such. - Scott made changes to the config and we can now run 3 ValidatesRunner.Dataflow in parallel (each run is about 2 hours) - With the latest quota changes, we peaked at ~70%

Re: Invite to comment on the @RequiresStableInput design doc

2018-07-03 Thread Lukasz Cwik
Does it make sense to only have some inputs be stable for a transform or for the entire transform to require stable inputs? On Tue, Jul 3, 2018 at 7:34 AM Kenneth Knowles wrote: > Since we always assume ProcessElement could have arbitrary side effects > (esp. randomization), the state and timers

Re: Invite to comment on the @RequiresStableInput design doc

2018-07-03 Thread Kenneth Knowles
Since we always assume ProcessElement could have arbitrary side effects (esp. randomization), the state and timers set up by a call to ProcessElement cannot be considered stable until they are persisted. It seems very similar to the cost of outputting to a downstream @RequiresStableInput transform,

Re: BiqQueryIO.write and Wait.on

2018-07-03 Thread Eugene Kirpichov
Awesome!! Thanks for the heads up, very exciting, this is going to make a lot of people happy :) On Tue, Jul 3, 2018, 3:40 AM Carlos Alonso wrote: > + dev@beam.apache.org > > Just a quick email to let you know that I'm starting developing this. > > On Fri, Apr 20, 2018 at 10:30 PM Eugene Kirpich

Re: BiqQueryIO.write and Wait.on

2018-07-03 Thread Carlos Alonso
+ dev@beam.apache.org Just a quick email to let you know that I'm starting developing this. On Fri, Apr 20, 2018 at 10:30 PM Eugene Kirpichov wrote: > Hi Carlos, > > Thank you for expressing interest in taking this on! Let me give you a few > pointers to start, and I'll be happy to help everywh

Re: Help! Beam SQL needs more committer support

2018-07-03 Thread James
Count me in, assign the PR to me when it is appropriate. On Tue, Jul 3, 2018 at 4:49 PM Ismaël Mejía wrote: > It has been long time since the last time I checked at the SQL work but in > case you guys still need more people to review you can count on me too. > > On Mon, Jul 2, 2018 at 7:23 PM Ru

Re: Help! Beam SQL needs more committer support

2018-07-03 Thread Ismaël Mejía
It has been long time since the last time I checked at the SQL work but in case you guys still need more people to review you can count on me too. On Mon, Jul 2, 2018 at 7:23 PM Rui Wang wrote: > Haha! I don't even know who I should @ now because there are so many > helpful hands! > > -Rui > > O

Re: [ANN] Apache Beam 2.5.0 has been released!

2018-07-03 Thread Jason Kuster
Excellent news, thank you JB and everyone else who helped out with the release! On Tue, Jul 3, 2018 at 12:00 AM Etienne Chauchot wrote: > Nice! > > Thanks for all the work you did on the release JB ! > > Etienne > > Le dimanche 01 juillet 2018 à 06:27 +0200, Jean-Baptiste Onofré a écrit : > > Th

Re: [ANN] Apache Beam 2.5.0 has been released!

2018-07-03 Thread Etienne Chauchot
Nice! Thanks for all the work you did on the release JB ! Etienne Le dimanche 01 juillet 2018 à 06:27 +0200, Jean-Baptiste Onofré a écrit : > The Apache Beam team is pleased to announce the release of 2.5.0 version! > > You can download the release here: > > https://beam.apache.org/get-s