Hey Tom,
> questions:Can my StreamTask receive a call when a checkpoint takes place
so that it can write its state?
The pattern for this is to take control of the checkpointing in your task,
rather than using task.commit.ms. If you:
# Set task.commit.ms=-1
# Set task.window.ms=60000
# Implement WindowableTask
Your task will have a window() method that receives a TaskCoordinator. You
can use the TaskCoordinator to call commit(). In this way, you have full
control over when your task commits, and can manage the committing of your
state in conjunction with that. For example, you could do:
window(...) {
state.commit()
coordinator.commit()
}
This would guarantee that you commit all external state before committing
your checkpoints (guaranteeing that you never lose data).
Cheers,
Chris
On Fri, Feb 6, 2015 at 7:11 AM, Thomas Bernhardt <
bernhardt...@yahoo.com.invalid> wrote:
> I would like to aggregate in-memory and when a checkpoint takes place,
> write the aggregation state someplace. I therefore have a few questions:Can
> my StreamTask receive a call when a checkpoint takes place so that it can
> write its state?Or can the CheckpointManager receive the task for which to
> checkpoint and the CheckpointManager can write the StreamTask state?I'm
> planning to have my external state also keep the topic+partition offsets in
> the same place that holds the aggregation state.
>
> Thank you, Best regards,Tom
>
>
>
