The current aim is the first option as you have correctly derived. :)
On May 1, 2015 5:39 PM, "Aljoscha Krettek" wrote:
> From this discussion I derive that we will have a state abstraction that
> everyone who requires state will work with? Or will the state be in object
> fields and they will be
>From this discussion I derive that we will have a state abstraction that
everyone who requires state will work with? Or will the state be in object
fields and they will be saved upon invocation of some doBackup() method.
On Apr 30, 2015 10:31 PM, "Stephan Ewen" wrote:
> That would be one way of
That would be one way of doing it, yes...
On Thu, Apr 30, 2015 at 10:23 PM, Gyula Fóra wrote:
> Okay, so the commit would be something like:
>
> commitState(OperatorState state)
>
>
> On Thu, Apr 30, 2015 at 10:17 PM, Stephan Ewen wrote:
>
> > I think your assumption (and the current kafka sour
Okay, so the commit would be something like:
commitState(OperatorState state)
On Thu, Apr 30, 2015 at 10:17 PM, Stephan Ewen wrote:
> I think your assumption (and the current kafka source implementation) is
> that there is one state object that you update/mutate all the time.
>
> If you draw a
I think your assumption (and the current kafka source implementation) is
that there is one state object that you update/mutate all the time.
If you draw a snapshot state object at the time of checkpoint, the source
can continue and that particular offset is remembered as the state of this
checkpoi
Regarding the commits (for instance kafka offset):
I dont exactly get how you mean to do this, if the source continues
processing after the checkpoint and before the commit, it will not know
what state has been committed exactly, so it would need to know the time of
checkpoint and store a local co
Thanks for the comments!
Concerning acknowledging the checkpoint:
The sinks need to definitely acknowledge it.
If we asynchronously write the state of operator (and emit downstream
barriers before that is complete),
then I think that we also need those operators to acknowledge the
checkp
I agree with all suggestions, thanks for summing it up Stephan.
A few more points I have in mind at the moment:
- Regarding the acknowledgements, indeed we don’t need to make all operators
commit back, we just have to make sure that all sinks have acknowledged a
checkpoint to consider it comple
