Sophie Blee-Goldman created KAFKA-10249:
-------------------------------------------
Summary: In-memory stores are skipped when checkpointing but not
skipped when reading the checkpoint
Key: KAFKA-10249
URL: https://issues.apache.org/jira/browse/KAFKA-10249
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: Sophie Blee-Goldman
Assignee: Sophie Blee-Goldman
As the title suggests, offsets for in-memory stores (including the suppression
buffer) are not written to the checkpoint file. However, when reading from the
checkpoint file during task initialization, we do not check
StateStore#persistent. We attempt to look up the offsets for in-memory stores
in the checkpoint file, and obviously do not find them.
With eos we have to conclude that the existing state is dirty and thus throw a
TaskCorruptedException. So pretty much any task with in-memory state will
always hit this exception when reinitializing from the checkpoint, forcing it
to clear the entire state directory and build up all of its state again from
scratch (both persistent and in-memory).
This is especially unfortunate for KIP-441, as we will hit this any time a task
is moved from one thread to another.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
