Re: checkpointing a OneInputStreamOperator/AbstractStreamOperator

2016-09-14 Thread Stefan Richter
Hi, I think it is still perfectly possible to implement something along the lines of the pull request. The major change in the current version is the introduction of key groups which are like prefixes to a key and are also functionally dependent on the key. Best, Stefan On Wed, Sep 14, 2016 at 1

Re: checkpointing a OneInputStreamOperator/AbstractStreamOperator

2016-09-14 Thread Aljoscha Krettek
Hi, yes, this observation is correct. It's a current limitation in Flink that I was trying to address with this issue https://issues.apache.org/jira/browse/FLINK-3582 and the PR I created for it. By now, that PR is quite outdated but we should maybe put in the effort to finish this now. +Stefan I'

Re: checkpointing a OneInputStreamOperator/AbstractStreamOperator

2016-09-13 Thread dan bress
Aljoscha, Thanks. I originally looked at the keyed state streaming. The problem I have with this approach is that it requires that I manage which keys this operator has seen and set them prior to getting or retrieving state, correct? My operator is used on a keyed stream. This means I have t

Re: checkpointing a OneInputStreamOperator/AbstractStreamOperator

2016-09-12 Thread Aljoscha Krettek
Hi, yes you guessed correctly: CheckpointedAsynchronously only works with functions and not with the lower-level StreamOperator. You would have to implement snapshotOperatorState() and restoreState(). These interfaces are quite low-level, though, and not stable. For example, in Flink 1.2 we're refa