Matthias J. Sax created KAFKA-21396:
---------------------------------------
Summary: Kafka Streams should report active task lag when restore
status changes
Key: KAFKA-21396
URL: https://issues.apache.org/jira/browse/KAFKA-21396
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Matthias J. Sax
Fix For: 4.5.0, 4.4.1
Kafka Streams using "streams" protocol reports task offsets for active
restoring tasks as well as standby/warmup tasks. When an active task is
RUNNING, no offsets are reported. This allows the
broker/group-coordinator/refiner to distinguish between active-restoring and
active-running tasks.
However, when an active-restoring task transits from RESTORING to RUNNING, no
immediate task-offset-push is triggered, but it only happens on the next
reporting interval (task.offset.interval.ms, default 60 sec). If a rebalance
happens during this time, the GC might make an incorrect decision base on stale
data).
For warmup-task, the client also reports task offsets on
`task.offset.interval.ms` during regular restore, but if a warmup task becomes
"hot" it report on every heartbeat. We should do the same thing for
active-restoring task, triggering a task-offset report when the
active-restoring task becomes "hot" and additionally, when is transits to
RUNNING.
While we cannot close the race condition entirely, as we are still subject to
the heartbeat interval (default: 5sec), it's still a significant improvement.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
