[
https://issues.apache.org/jira/browse/KAFKA-15116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750311#comment-17750311
]
David Gammon commented on KAFKA-15116:
--------------------------------------
# The store is storing everything from all partitions which does have it's
limitations.
# It is blocking because it knows that a message has been sent to the topic
but not committed. The idea is that the store blocks until all messages it
knows about have been sent and been committed and therefore populated into the
store. It's a way of making sure the store is up to date before information is
read from it.
# We aren't touching the change log topic.
# It is the behaviour we have observed and validated with a fix to the core
streams code.
# Committing the open transactions is fine (if possible). The problem is the
un committed transactions due to the rebalancing. If we have two un committed
transactions for the same partition key we end up in the blocking state because
the second message cannot be processed because the first message hasn't been
committed. The old behaviour sounds like it would solve our problem. Is there a
configuration option to switch this back on?
# To answer your question "How should the system know if there is a
dependency?": Through configuration. I don't think anything that we are trying
to do is going against how Kafka is designed. It might be non optimal and
legacy but it does feel like something that streams should be flexible enough
to handle. Why can't we chose to "stop the world"?
> Kafka Streams processing blocked during rebalance
> -------------------------------------------------
>
> Key: KAFKA-15116
> URL: https://issues.apache.org/jira/browse/KAFKA-15116
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.5.0
> Reporter: David Gammon
> Priority: Major
>
> We have a Kafka Streams application that simply takes a messages, processes
> it and then produces an event out the other side. The complexity is that
> there is a requirement that all events with the same partition key must be
> committed before the next message is processed.
> This works most of the time flawlessly but we have started to see problems
> during deployments where the first message blocks the second message during a
> rebalance because the first message isn’t committed before the second message
> is processed. This ultimately results in transactions timing out and more
> rebalancing.
> We’ve tried lots of configuration to get the behaviour we require with no
> luck. We’ve now put in a temporary fix so that Kafka Streams works with our
> framework but it feels like this might be a missing feature or potentially a
> bug.
> +Example+
> Given:
> * We have two messages (InA and InB).
> * Both messages have the same partition key.
> * A rebalance is in progress so streams is no longer able to commit.
> When:
> # Message InA -> processor -> OutA (not committed)
> # Message InB -> processor -> blocked because #1 has not been committed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
