Re: checkpoint state keeps on increasing

2016-08-18 Thread Janardhan Reddy
I also thought that the checkpointing state size was growing due to growing key space, i registered processing time timer on 'onevent' and wrote a custom trigger and still the checkpointing state size was growing. Our code is linked with flink 1.0.0 jar but was running on flink 1.1.1 (yarn session

Re: checkpoint state keeps on increasing

2016-08-18 Thread Stephan Ewen
Hi! Count windows are a little tricky. If you have a growing key space, the state keeps growing. Lets say you have a bunch of values for key "A". You will fire the count windows for every two elements and keep one element. If "A" never comes again after that, the one element will still be kept ar

checkpoint state keeps on increasing

2016-08-17 Thread Janardhan Reddy
Hi, I am noticing that the checkpointing state has been constantly growing for the below subtask. Only the current active window elements should be checkpointed ? why is it constantly growing ? finalStream.keyBy("<>").countWindow(2,1) .apply((_, _, input: scala.Iterable[], out: Collector[]) =>