Thanks for the update! Let us know if you need anything. On Fri, 4 Nov 2016 at 20:43 Maciek Próchniak <m...@touk.pl> wrote:
> Hi Aljoscha, > > I know it's tricky... > > Few weeks ago we decided to implement it without windows, using just > stateful operator and some queues/map per key as state - so yeah, we tried > to imagine how to do this in plain java and one stream ;) > > We also process watermarks to evict old events. Fortunately, our streams > are not that big and we can keep all "recent" events in state - without > preaggregation. > > Currently we're waiting for some feedback from our client on results - if > it's ok, we'll stick with that, otherwise we'll have to look into it > deeper... > > thanks, > > maciek > > > On 25/10/2016 16:41, Aljoscha Krettek wrote: > > Hi Maciek, > cases like this, where you essentially want to evict elements that are > older than a certain threshold while keeping a count of those elements that > are not older than that threshold tend to be quite tricky. > > In order to start thinking about this, how would you implement this case > in a non-parallel way, in plain Java. You have the stream of incoming > events, they possibly have timestamps, they are possibly not ordered by > that timestamp (this depends on your use case). Now, what are the > algorithms/data structures that could be used for computing the result that > you require? > > Cheers, > Aljoscha > > On Fri, 23 Sep 2016 at 10:50 Claudia Wegmann <c.wegm...@kasasi.de> wrote: > > Hey, > > I'm no expert at all, but for me this sounds like a use case for Complex > Event Processing (CEP). I don't know if you're aware of Flinks CEP Library > [1, 2]? Maybe that solves your problem of multiple firings. But best to > wait for the experts to answer your questions on handling state and firing > windows :) > > Best, > Claudia > > [1]: https://flink.apache.org/news/2016/04/06/cep-monitoring.html > [2]: > https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/cep.html > > > -----Ursprüngliche Nachricht----- > Von: Maciek Próchniak [mailto:m...@touk.pl] > Gesendet: Freitag, 23. September 2016 10:36 > An: user@flink.apache.org > Betreff: window-like use case > > Hi, > > in our project we're dealing with a stream of billing events. Each has > customerId and charge amount We want to have a process that will trigger > event (alarm) when sum of charges for customer during last 4 hours exceeds > certain threshold, say > - 10. > The triggered event should contain data from last billing event (the one > that triggered alarm) > > One one hand we can implement it as custom state - we'd save charges (or > some precomputed aggregates) from last 4 hours and trigger event when new > one arrives. > OTOH we've been thinking if we can make it easier by using flink windows. > We tried to model our situation as sliding events (of length 4hours, slide > is 1h), have some precomputed aggregate and have a custom trigger that > fires on element when threshold is exceeded. > It kinda works, except for the fact that state is unnecesarily large, > custom trigger is a bit hacky and (worst of all) when event with charge > amount e.g. 20 arrives trigger fires in all slides and we have duplicated > events. > That's why we currently think about implementing it by custom state... > > Do you have any other ideas/recommendations how can we handle such > requirement? > > thanks, > maciek > > >
