On Fri, May 20, 2016 at 3:12 PM, Abhishek Singh
wrote:
> Thanks. I am still in theory/evaluation mode. Will try to code this up to
> see if checkpoint will become an issue. I do have a high rate of ingest and
> lots of in flight data. Hopefully flink back pressure keeps this nicely
> bounded.
Sur
Thanks. I am still in theory/evaluation mode. Will try to code this up to
see if checkpoint will become an issue. I do have a high rate of ingest and
lots of in flight data. Hopefully flink back pressure keeps this
nicely bounded.
I doubt it will be a problem for me - because even spark is writing
On Thu, May 19, 2016 at 7:48 PM, Abhishek R. Singh
wrote:
> There seems to be some relationship between watermarks, triggers and
> checkpoint that is someone not being leveraged.
Checkpointing is independent of this, yes. Did the state size become a
problem for your use case? There are various us
Thanks - appreciate the response.
The reason I want to control these things is this - my state grows and shrinks
over time (user level windowing as application state). I would like to trigger
checkpoints just after the state has been crunched/compressed (at the window
boundary). Say I crunch ev
Hi Abhishek,
you can implement custom sources by implementing the SourceFunction or the
ParallelSourceFunction interface and then calling
StreamExecutionEnvironment.addSource.
At the moment, it is not possible to control manually or from a source
function when to trigger a checkpoint. This is the
