Watermarks as "process completion" flags

2015-11-30 Thread Anton Polyakov
gt;> >>> Have a look here for an illustration about barriers flowing with the >>> stream: >>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/stream_checkpointing.html >>> >>> Stephan >>> >>> >>> On Mon, No

Re: Watermarks as "process completion" flags

2015-11-30 Thread Anton Polyakov
to "checkpointComplete()" in the sources comes > after all barriers have reached all sinks. > > Have a look here for an illustration about barriers flowing with the > stream: > https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/stream_checkpointing.html &

Re: Working with State example /flink streaming

2015-11-30 Thread Anton Polyakov
Javier sorry to jumping in, but I think your case is very similar to what I am trying to achieve in the thread just next to yours (called "Watermarks as "process completion" flags". I also need to process a stream which is produced for some time, but then take an action after certain event. Also w

Re: Watermarks as "process completion" flags

2015-11-30 Thread Anton Polyakov
once they have > flown all the way through. > > Greetings, > Stephan > > > On Mon, Nov 30, 2015 at 11:03 AM, Anton Polyakov > wrote: > >> I think I can turn my problem into a simpler one. >> >> Effectively what I need - I need way to checkpoint certain ev

Re: Watermarks as "process completion" flags

2015-11-30 Thread Anton Polyakov
once they have > flown all the way through. > > Greetings, > Stephan > > > On Mon, Nov 30, 2015 at 11:03 AM, Anton Polyakov > wrote: > >> I think I can turn my problem into a simpler one. >> >> Effectively what I need - I need way to checkpoint certain ev

Re: Watermarks as "process completion" flags

2015-11-30 Thread Anton Polyakov
processed. On Sun, Nov 29, 2015 at 4:12 PM, Anton Polyakov wrote: > Hi Fabian > > Defining a special flag for record seems like a checkpoint barrier. I > think I will end up re-implementing checkpointing myself. I found the > discussion in flink-dev: > mail-archives.apache.org/mo

Re: Watermarks as "process completion" flags

2015-11-29 Thread Anton Polyakov
ther the > elements in the window should be retained or removed after the evaluation. > > Best, Fabian > > > > 2015-11-24 21:32 GMT+01:00 Anton Polyakov <mailto:polyakov.an...@gmail.com>>: > Hi Max > > thanks for reply. From what I understand wind

Re: Watermarks as "process completion" flags

2015-11-24 Thread Anton Polyakov
> > Thanks, > Max > > On Sun, Nov 22, 2015 at 8:53 PM, Anton Polyakov > wrote: > > Hi > > > > I am very new to Flink and in fact never used it. My task (which I > currently solve using home grown Redis-based solution) is quite simple - I > have a system

Watermarks as "process completion" flags

2015-11-22 Thread Anton Polyakov
Hi I am very new to Flink and in fact never used it. My task (which I currently solve using home grown Redis-based solution) is quite simple - I have a system which produces some events (trades, it is a financial system) and computational chain which computes some measure accumulatively over th