[
https://issues.apache.org/jira/browse/KAFKA-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-14641:
----------------------------------
Component/s: streams
> Cleanup CommitNeeded after EOS-V1 is removed
> --------------------------------------------
>
> Key: KAFKA-14641
> URL: https://issues.apache.org/jira/browse/KAFKA-14641
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Guozhang Wang
> Priority: Major
>
> This is a follow-up of KAFKA-14294.
> Today we have several flags to determine if KS need to execute a commit: 1)
> task-level "commitNeeded" which is set whenever process() or punctuator() is
> called, 2) if there are input topic offsets to commit, retrieved from the
> "task.prepareCommit()", 3) the "transactionInFlight" flag from producer as a
> fix of KAFKA-14294 (this subsumes the first "commitNeeded" functionality).
> Given that we are still having EOS-v1, cleanup this would be a bit complex.
> But after the deprecated EOS-V1 is removed, we can cleanup those controls
> since for any commit cases, we would need to commit all tasks anyways whereas
> in EOS-v1, we would commit probably a subset of tasks since they are done by
> different producers and hence different txns.
> A quick thought is the following:
> 1) We would not need the per-task "commitNeeded" anymore.
> 2) We would maintain a single "commitNeeded" flag on the task-executor, hence
> on the thread level. It is set whenever `process()` or `punctuator` is called.
> 3) Whenever we need to commit, either a) periodically, b) upon revocation, c)
> upon user request, we simply check that flag, and if necessary commit all
> tasks and reset the flag.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
