Re: State snapshotting when source is finite

2017-10-26 Thread Flavio Pompermaier
Done: https://issues.apache.org/jira/browse/FLINK-7930 Best, Flavio On Thu, Oct 26, 2017 at 10:52 AM, Till Rohrmann wrote: > Hi Flavio, > > this kind of feature is indeed useful and currently not supported by > Flink. I think, however, that this feature is a bit trickier to implement, > because

Re: State snapshotting when source is finite

2017-10-26 Thread Till Rohrmann
Hi Flavio, this kind of feature is indeed useful and currently not supported by Flink. I think, however, that this feature is a bit trickier to implement, because Tasks cannot currently initiate checkpoints/savepoints on their own. This would entail some changes to the lifecycle of a Task and an e

Re: State snapshotting when source is finite

2017-10-26 Thread Fabian Hueske
Hi Flavio, Thanks for bringing up this topic. I think running periodic jobs with state that gets restored and persisted in a savepoint is a very valid use case and would fit the stream is a superset of batch story quite well. I'm not sure if this behavior is already supported, but think this would

State snapshotting when source is finite

2017-10-25 Thread Flavio Pompermaier
Hi to all, in my current use case I'd like to improve one step of our batch pipeline. There's one specific job that ingest a tabular dataset (of Rows) and explode it into a set of RDF statements (as Tuples). The objects we output are a containers of those Tuples (grouped by a field). Flink statefu