Re: Windowing Guarantees in samza

2015-02-16 Thread Benjamin Edwards
Thanks for the responses, much appreciated. I will continue to experiment. Ben On Sun Feb 15 2015 at 21:22:17 Julian Hyde wrote: > +1 > > As far as possible, behavior should be deterministic, that is, determined > by the data rather than when the query was started or the arrival time of > the d

Re: Windowing Guarantees in samza

2015-02-15 Thread Julian Hyde
+1 As far as possible, behavior should be deterministic, that is, determined by the data rather than when the query was started or the arrival time of the data. Of course, for the query to make progress, there should be ways to discard late data and to indicate that a producer is alive but do

Re: Windowing Guarantees in samza

2015-02-15 Thread Karthik Sriram
Hi Benjamin, I'm trying to employ samza for a similar use case and this is what I did to mitigate this: 1> I have a notion of timestamp in the messages itself that I listen to. This way, as I get messages, I can maintain state by time period of aggregation by attaching the time period to the key

Windowing Guarantees in samza

2015-02-15 Thread Benjamin Edwards
Hi Based on what I can see in the run loop class, there are a few things that seem a little problematic for windowed processing with respect to time: 1) No ability to schedule *when* on an interval you might start. For instance, if you wanted to process a window on the hour, every hour, there is