Re: Proposal on porting PAssert away from aggregators

2017-03-30 Thread Dan Halperin
Yeah, this sounds really nice. It also seems to let runners do "whatever they want" -- write to files, switch to some magic but small-scale state, etc. On Thu, Mar 30, 2017 at 10:50 PM, Jean-Baptiste Onofré wrote: > Hi Pablo, > > it sounds a good idea ! > > Regards > JB > > > On 03/31/2017 12:12

Re: Proposal on porting PAssert away from aggregators

2017-03-30 Thread Jean-Baptiste Onofré
Hi Pablo, it sounds a good idea ! Regards JB On 03/31/2017 12:12 AM, Pablo Estrada wrote: Hello all, BEAM-775 aims to remove Aggregators from the Java SDK. One of the blocking issues is BEAM-1148, porting PAssert away from using them. Let me summarize the issues that we should consider; and t

Re: [PROPOSAL] @OnWindowExpiration

2017-03-30 Thread Jean-Baptiste Onofré
+1 Regards JB On 03/28/2017 10:27 PM, Kenneth Knowles wrote: Hi all, I have a little extension to the stateful DoFn annotations to circulate for feedback: Allow a method to be annotated with @OnWindowExpiration to automatically get a callback at some point after the window has expired, but bef

Re: Apache Beam - JIRA Contributor Role

2017-03-30 Thread Rahul Sabbineni
Thank you! On Fri, Mar 31, 2017 at 1:34 AM, Jean-Baptiste Onofré wrote: > Done, > > Looking forward your contributions ! > > Thanks ! > Regards > JB > > > On 03/31/2017 07:30 AM, Rahul Sabbineni wrote: > >> Hey everyone, >> >> I've found the Beam project delightful, and I'd love to help by picki

Re: Apache Beam - JIRA Contributor Role

2017-03-30 Thread Jean-Baptiste Onofré
Done, Looking forward your contributions ! Thanks ! Regards JB On 03/31/2017 07:30 AM, Rahul Sabbineni wrote: Hey everyone, I've found the Beam project delightful, and I'd love to help by picking off some of the lower priority issues to start. Would it be possible to obtain a JIRA contributo

Apache Beam - JIRA Contributor Role

2017-03-30 Thread Rahul Sabbineni
Hey everyone, I've found the Beam project delightful, and I'd love to help by picking off some of the lower priority issues to start. Would it be possible to obtain a JIRA contributor role in order to assign the following issue (https://issues.apache.org/jira/browse/BEAM-1684) to myself? My full

Re: [PROPOSAL] @OnWindowExpiration

2017-03-30 Thread Kenneth Knowles
On Thu, Mar 30, 2017 at 6:17 PM, Tyler Akidau wrote: > (i.e., or do we disallow user-specified timers for merging windows > currently, similar to value state)? > We currently disallow. They'll need a combiner on the timestamps to be merged. Kenn > -Tyler > > On Wed, Mar 29, 2017 at 2:38 PM A

Re: [PROPOSAL] @OnWindowExpiration

2017-03-30 Thread Tyler Akidau
+1. I'm assuming this is much easier for the user in the case of merging windows (i.e., or do we disallow user-specified timers for merging windows currently, similar to value state)? -Tyler On Wed, Mar 29, 2017 at 2:38 PM Aljoscha Krettek wrote: > +1 I had also already commented on the issue a

Re: Proposal on porting PAssert away from aggregators

2017-03-30 Thread Kenneth Knowles
On Thu, Mar 30, 2017 at 3:42 PM, Ben Chambers wrote: > When you say instance do you mean a different named PTransform class? Or > just a separate instance of whatever "HandleAssertResult" PTransform is > introduced? If the latter, I believe that makes sense. The latter. One class that "counts t

Re: Proposal on porting PAssert away from aggregators

2017-03-30 Thread Ben Chambers
When you say instance do you mean a different named PTransform class? Or just a separate instance of whatever "HandleAssertResult" PTransform is introduced? If the latter, I believe that makes sense. It also addresses the problems with committed vs. attempted metrics. Specifically: Since we want t

Re: Proposal on porting PAssert away from aggregators

2017-03-30 Thread Kenneth Knowles
This sounds pretty good to me. I really like emphasizing the ability of a runner to go ahead and use pipeline surgery to put in their own verification transform. On Thu, Mar 30, 2017 at 3:12 PM, Pablo Estrada wrote: > * The current asserting DoFns in PAssert will go from being DoFn Void> to be D

Proposal on porting PAssert away from aggregators

2017-03-30 Thread Pablo Estrada
Hello all, BEAM-775 aims to remove Aggregators from the Java SDK. One of the blocking issues is BEAM-1148, porting PAssert away from using them. Let me summarize the issues that we should consider; and then propose a solution that seems reasonable: 1. We want the test Pipeline run to fail-fast in

Re: [DISCUSSION] rename StateSpecs.combiningValue?

2017-03-30 Thread Kenneth Knowles
+1 for a different reason (also: now is the time to revisit names and other bits of the State API before it is too late :-) Folks may not have this catalog in their head. The classes / methods are: ValueState / value(...) BagState / bag(...) SetState / set(...) MapState / map(...) AccumulatorComb

Re: [DISCUSSION] rename StateSpecs.combiningValue?

2017-03-30 Thread Jean-Baptiste Onofré
+0 as the StateSpec takes AccumularCombiningState, it's already "specify" indirectly. Regards JB On 03/30/2017 04:46 PM, Etienne Chauchot wrote: Hi all, Just a 10 cents comment, but maybe a rename of public method nevertheless... There are AccumulatorCombiningState and CombiningState interf

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

2017-03-30 Thread Amit Sela
I will not be able to make it this weekend, too busy. Let's chat at the beginning of next week and see what's on my plate. On Tue, Mar 28, 2017 at 5:44 PM Aljoscha Krettek wrote: > Thanks for the offers, guys! The code is finished, though. I only need > to do the last touch ups. > > On Tue, Mar

[DISCUSSION] rename StateSpecs.combiningValue?

2017-03-30 Thread Etienne Chauchot
Hi all, Just a 10 cents comment, but maybe a rename of public method nevertheless... There are AccumulatorCombiningState and CombiningState interfaces, the first extends the second The factory method StateSpecs.combiningValue returns a StateSpec ** > The method name seems confusing for a u