[
https://issues.apache.org/jira/browse/KAFKA-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18040938#comment-18040938
]
Stef Noten commented on KAFKA-19593:
------------------------------------
I have encountered the same issue on a single node installation (single Kafka
container, with both broker and controller roles).
We took a backup of the production data dir and can consistently reproduce the
issue when starting a new container from that data.
* We had an unclean shutdown and a transaction involving __consumer_offsets-49
in PREPARE_ABORT
** Before starting on that data dir backup, the last metadata snapshot shows
the following for __consumer_offsets-49:
{code:java}
PartitionRegistration(replicas=[0], directories=[AAAAAAAAAAAAAAAAAAAAAA],
isr=[0], removingReplicas=[], addingReplicas=[], elr=[], lastKnownElr=[],
leader=0, leaderRecoveryState=RECOVERED, leaderEpoch=6, partitionEpoch=6){code}
* At startup, the controller and broker both are able to successfully start
** It's started with an older epoch:
{code:java}
[2025-11-26 10:17:11,144] INFO Initialized snapshots with IDs
SortedSet(OffsetAndEpoch[offset=12656, epoch=3], OffsetAndEpoch[offset=19855,
epoch=3], OffsetAndEpoch[offset=27054, epoch=3], OffsetAndEpoch[offset=34253,
epoch=3], OffsetAndEpoch[offset=41452, epoch=3], OffsetAndEpoch[offset=48651,
epoch=3], OffsetAndEpoch[offset=55850, epoch=3], OffsetAndEpoch[offset=63048,
epoch=3], OffsetAndEpoch[offset=70246, epoch=3], OffsetAndEpoch[offset=77445,
epoch=3], OffsetAndEpoch[offset=84644, epoch=3], OffsetAndEpoch[offset=91843,
epoch=3], OffsetAndEpoch[offset=99042, epoch=3], OffsetAndEpoch[offset=106241,
epoch=3], OffsetAndEpoch[offset=113440, epoch=3], OffsetAndEpoch[offset=120639,
epoch=3], OffsetAndEpoch[offset=127837, epoch=3], OffsetAndEpoch[offset=135036,
epoch=3], OffsetAndEpoch[offset=142235, epoch=3], OffsetAndEpoch[offset=149434,
epoch=3], OffsetAndEpoch[offset=156632, epoch=3], OffsetAndEpoch[offset=163830,
epoch=3], OffsetAndEpoch[offset=171875, epoch=3], OffsetAndEpoch[offset=179070,
epoch=3], OffsetAndEpoch[offset=186265, epoch=3], OffsetAndEpoch[offset=193461,
epoch=3], OffsetAndEpoch[offset=200657, epoch=3], OffsetAndEpoch[offset=207853,
epoch=3], OffsetAndEpoch[offset=215049, epoch=3], OffsetAndEpoch[offset=222245,
epoch=3], OffsetAndEpoch[offset=229441, epoch=3], OffsetAndEpoch[offset=236637,
epoch=3], OffsetAndEpoch[offset=243833, epoch=3], OffsetAndEpoch[offset=251029,
epoch=3], OffsetAndEpoch[offset=258225, epoch=3], OffsetAndEpoch[offset=265421,
epoch=3], OffsetAndEpoch[offset=272617, epoch=3], OffsetAndEpoch[offset=279813,
epoch=3], OffsetAndEpoch[offset=287009, epoch=3]) from
/var/lib/kafka/data/__cluster_metadata-0 (kafka.raft.KafkaMetadataLog$)
...
[2025-11-26 10:17:11,150] INFO [RaftManager id=0] Loading snapshot
(OffsetAndEpoch[offset=287009, epoch=3]) since log start offset (0) is greater
than the internal listener's next offset (-1)
(org.apache.kafka.raft.internals.KRaftControlRecordStateMachine){code}
** After the broker reaches the RUNNING state, the half-completed transaction
is attempted to transition to COMPLETE_ABORT:
{code:java}
[Transaction State Manager 0]: Sending txn markers for
TransactionalIdCoordinatorEpochAndTransitMetadata(processing-v3-d74cb07b-583b-4dac-8150-7f4e5061e46a-2,8,ABORT,TransactionMetadata(transactionalId=processing-v3-d74cb07b-583b-4dac-8150-7f4e5061e46a-2,
producerId=6, prevProducerId=-1, nextProducerId=-1, producerEpoch=3,
lastProducerEpoch=-1, txnTimeoutMs=10000, state=PREPARE_ABORT,
pendingState=Some(COMPLETE_ABORT),
topicPartitions=HashSet(logical.sensor-data-1, logical.sensor-data-0,
__consumer_offsets-49), txnStartTimestamp=1762465875834,
txnLastUpdateTimestamp=1762465904909,
clientTransactionVersion=TV_0),TxnTransitMetadata[producerId=6,
prevProducerId=6, nextProducerId=-1, producerEpoch=3, lastProducerEpoch=-1,
txnTimeoutMs=10000, txnState=COMPLETE_ABORT, topicPartitions=[],
txnStartTimestamp=1762465875834, txnLastUpdateTimestamp=1764148665796,
clientTransactionVersion=TV_0]) after loading partition 22
(kafka.coordinator.transaction.TransactionStateManager) {code}
** Which ends up in an eternal loop on following error:
{code:java}
Sending processing-v3-d74cb07b-583b-4dac-8150-7f4e5061e46a-2's transaction
marker for partition __consumer_offsets-49 has failed with error
org.apache.kafka.common.errors.NotLeaderOrFollowerException, retrying with
current coordinator epoch 8
(kafka.coordinator.transaction.TransactionMarkerRequestCompletionHandler) {code}
* I don't need to start my Kafka Streams app to trigger this error loop (but
it indeed does not start successfully when attempted)
In production, my customer preferred availability over retaining its data so
they wiped Kafka and reconfigured the system. Removal of the __consumer_offsets
partition seems slightly better but it's still problematic. Meanwhile I've been
able to circumvent the error loop on the data dir backup by modifying the
broker id, but from my understanding this is also not a safe option.
Other attempts:
* kafka-leader-election.sh, this did not help:
**
{code:java}
Valid replica already elected for partitions __consumer_offsets-49 {code}
> Stuck __consumer_offsets partition (kafka streams app)
> ------------------------------------------------------
>
> Key: KAFKA-19593
> URL: https://issues.apache.org/jira/browse/KAFKA-19593
> Project: Kafka
> Issue Type: Bug
> Components: consumer, streams
> Affects Versions: 4.0.0
> Reporter: Matej Pucihar
> Priority: Major
> Labels: kafka-streams
>
> h3. Problem Summary
> My Kafka Streams application cannot move its {{state_store}} from
> {{STARTING}} to {{{}RUNNING{}}}.
> I'm using a *Strimzi Kafka cluster* with:
> * 3 *controller nodes*
> * 4 *broker nodes*
> h3. Observations
> h4. Partition {{__consumer_offsets-35}} is {*}stuck{*}.
> From AKHQ, partition details:
> * *Broker 10* is the *leader* of {{__consumer_offsets-35}}
> * There are *no interesting logs* on broker 10
> * However, logs are *spamming every 10ms* from broker 11 (a {*}replica{*}):
> 2025-08-11 04:05:50 INFO [TxnMarkerSenderThread-11]
> TransactionMarkerRequestCompletionHandler:66
> [Transaction Marker Request Completion Handler 10]: Sending
> irm_r_sbm2_web-backend-web-1-6cad35c7-2be9-4ed2-9849-9c059cc8c409-4's
> transaction marker for partition __consumer_offsets-35 has failed with error
> org.apache.kafka.common.errors.NotLeaderOrFollowerException, retrying with
> current coordinator epoch 38
> h4. Brokers 20 and 21 — neither leaders nor replicas — also spamming the same
> error:
> *Broker 20:*
> 2025-08-11 04:39:45 INFO [TxnMarkerSenderThread-20]
> TransactionMarkerRequestCompletionHandler:66
> Sending irm_r_sbm2_web-backend-web-1-6cad35c7-2be9-4ed2-9849-9c059cc8c409-3's
> transaction marker for partition __consumer_offsets-35 has failed with error
> org.apache.kafka.common.errors.NotLeaderOrFollowerException, retrying with
> current coordinator epoch 54
>
> *Broker 21:*
> 2025-08-11 04:39:58 INFO [TxnMarkerSenderThread-21]
> TransactionMarkerRequestCompletionHandler:66
> Sending irm_r_sbm2_web-backend-web-1-6cad35c7-2be9-4ed2-9849-9c059cc8c409-2's
> transaction marker for partition __consumer_offsets-35 has failed with error
> org.apache.kafka.common.errors.NotLeaderOrFollowerException, retrying with
> current coordinator epoch 28
>
> ----
> h3. Kafka Streams App Behavior
> Logs from the Kafka Streams app (at debug level) repeat continuously. The
> {{state_store}} *never transitions* from {{STARTING}} to {{{}RUNNING{}}}.
> Key repeated logs (debug log level):
> * Polling main consumer repeatedly
> * SASL/SCRAM authentication succeeds
> * 0 records fetched
> * 0 records processed
> * Punctuators run, but nothing gets committed
> * Fails to commit due to {*}rebalance in progress{*}, retrying…
> {{}}
> ----
> h3. Workarounds Considered
> The *only thing that temporarily resolves the issue* is:
> * Physically deleting the partition files for {{__consumer_offsets-35}} from
> both the leader and replica brokers
> Other drastic options:
> * Deleting the entire {{__consumer_offsets}} topic
> * Re-creating the entire Kafka cluster
> ----
> h3. Additional Info
> * I cannot reproduce this in a *clean git project*
> * The issue is isolated to a {*}"corrupt" cluster{*}, which is still
> available for inspection
> * This problem has occurred *4 times* in the *past month*
> * It *started happening after upgrading from Strimzi 3.9 to 4.0*
> * I'm using quarkus (kafka-stream version is 4.0.0) with default
> configuration, the only config worth mentioning is that I'm using
> exactly_once_v2 processing guarantee.
> ----
> h3. Help Needed
> I'm hoping someone can {*}make sense of this issue{*}.
> Please feel free to *reach out.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
