Hello, we are updating one of our clusters from version 2.0 to 2.2. The cluster has 4 brokers. After stopping the first broker the cluster was still operating normally as expected, receiving data from producers and sending data to consumers.
When starting the first broker again with the new version 2.2, the brokers start showing lots of the following messages: INFO [Transaction Marker Request Completion Handler 4]: Sending client-x's transaction marker for partition topic-name-1 has failed with error org.apache.kafka.common.errors.NotLeaderForPartitionException, retrying with current coordinator epoch 0 (kafka.coordinator.transaction.TransactionMarkerRequestCompletionHandler) ...and INFO [ReplicaFetcher replicaId=1, leaderId=4, fetcherId=8] Retrying leaderEpoch request for partition another-topic-1 as the leader reported an error: NOT_LEADER_FOR_PARTITION (kafka.server.ReplicaFetcherThread) In the end, all Brokers start showing logs for the transaction initialization: INFO [TransactionCoordinator id=1] Initialized transactionalId client-x with producerId 5388 and producer epoch 6 on partition __transaction_state-36 (kafka.coordinator.transaction.TransactionCoordinator) Meanwhile that happens (all the logs showing up) the cluster stop receiving data, it took around 2 hours for the cluster to be up and running normally. Then all our producers and consumers were able to work correctly. Once all Brokers stop showing constantly the logs about the Transaction initialization process, is when actually all producers start sending data gradually to the Brokers. Is that behavior expected? I mean, the fact that the cluster stop working as expected (receiving data from producers and sending data to consumers normally) Broker update from 2.0 to 2.2 Replication factor 4 Min in sync replicas 3 Producers version: 0.11.0.3 ~1100 Producers Exactly once semantic and transaction enabled in Producers. Consumer version: 2.2.0 ~64 Consumers I hope someone can give me a clue what could be happening, maybe some configuration to review. Thanks a lot in advance. -- Santilli Jonathan
