[
https://issues.apache.org/jira/browse/KAFKA-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622498#comment-17622498
]
A. Sophie Blee-Goldman commented on KAFKA-14294:
------------------------------------------------
Mm, yeah this sounds similar to what I found in
https://issues.apache.org/jira/browse/KAFKA-13295. Seems like we have a general
vulnerability to uncommitted transactions. Unfortunately with the nontrivial
overhead of committing, it's tricky to strike the right balance between keeping
the producers alive and over-committing to the point of serious performance
loss :/
Perhaps we need a better overall solution to the problem rather than playing
whack-a-mole as issues like this come up. Fortunately, I actually think our
work around decoupling the consumer from the processing thread(s) may naturally
fix this, if we get it right.
I'm still in talks with [~guozhang] about some of the design details, including
where the producers should fit into the new architecture. This is definitely
some good food for thought there, if we're smart about it hopefully we can
avoid any more issues like this with low(er) cost commits on a regular basis.
> Kafka Streams should commit transaction when no records are processed
> ---------------------------------------------------------------------
>
> Key: KAFKA-14294
> URL: https://issues.apache.org/jira/browse/KAFKA-14294
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 3.2.1
> Reporter: Vicky Papavasileiou
> Priority: Major
>
> Currently, if there are no records to process in the input topic, a
> transaction does not commit. If a custom punctuator code is writing to a
> state store (which is common practice) the producer gets fenced when trying
> to write to the changelog topic. This throws a TaskMigratedException and
> causes a rebalance.
> A better approach would be to commit a transaction even when there are no
> records processed as to allow the punctuator to make progress.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
