Requiring PTransform to set a coder on its resulting collections

2017-07-25 Thread Eugene Kirpichov
Hello, I've worked on a few different things recently and ran repeatedly into the same issue: that we do not have clear guidance on who should set the Coder on a PCollection: is it responsibility of the PTransform that outputs it, or is it responsibility of the user, or is it sometimes one and som

Re: Should Pipeline wait till all processing time timers fire before exit?

2017-07-25 Thread Eugene Kirpichov
Yes, and I think in this case the pipeline should never transition to DONE. On Tue, Jul 25, 2017 at 3:42 PM Robert Bradshaw wrote: > I generally agree, but it's unclear what to do with timers that are > scheduled during the execution of existing timers. (For example, a > "heartbeat" source may p

Re: Should Pipeline wait till all processing time timers fire before exit?

2017-07-25 Thread Robert Bradshaw
I generally agree, but it's unclear what to do with timers that are scheduled during the execution of existing timers. (For example, a "heartbeat" source may process a timer by emitting an element and scheduling a timer for the future. One would never be able to fire "all" timers. I suppose this is

Re: Towards a spec for robust streaming SQL, Part 2

2017-07-25 Thread Pramod Immaneni
Thanks for the invitation Tyler. I am sure folks who worked on the calcite integration and others would be interested. On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau wrote: > +d...@apex.apache.org, since I'm told Apex has a Calcite integration as > well. If anyone on the Apex side wants to join

Re: Towards a spec for robust streaming SQL, Part 2

2017-07-25 Thread Tyler Akidau
+d...@apex.apache.org, since I'm told Apex has a Calcite integration as well. If anyone on the Apex side wants to join in on the fun, your input would be welcomed! -Tyler On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau wrote: > Hello Flink, Calcite, and Beam dev lists! > > Linked below is the sec