Re: Using Flink watermarks and a large window state for scheduling

2016-06-09 Thread Josh
Ok, thanks Aljoscha. As an alternative to using Flink to maintain the schedule state, I could take the (e, t2) stream and write to a external key-value store with a bucket for each minute. Then have a separate service which polls the key-value store every minute and retrieves the current bucket, a

Re: Using Flink watermarks and a large window state for scheduling

2016-06-09 Thread Aljoscha Krettek
Hi Josh, I'll have to think a bit about that one. Once I have something I'll get back to you. Best, Aljoscha On Wed, 8 Jun 2016 at 21:47 Josh wrote: > This is just a question about a potential use case for Flink: > > I have a Flink job which receives tuples with an event id and a timestamp > (e

Using Flink watermarks and a large window state for scheduling

2016-06-08 Thread Josh
This is just a question about a potential use case for Flink: I have a Flink job which receives tuples with an event id and a timestamp (e, t) and maps them into a stream (e, t2) where t2 is a future timestamp (up to 1 year in the future, which indicates when to schedule a transformation of e). I