Hi Justine, Thank you for your quick reply! I created https://issues.apache.org/jira/browse/KAFKA-16692 to describe the issue. Happy to provide any additional information needed.
Regards, Johnson On Tue, 7 May 2024 at 23:18, Justine Olshan <jols...@confluent.io.invalid> wrote: > Hi Johnson, > > Thanks for bringing this issue to the mailing list. > > I'm familiar with the change you are referring to. However, during the > upgrade you should be hitting this code path and we should not sending > requests to older version brokers. > > https://github.com/apache/kafka/blob/525b9b1d7682ae2a527ceca83fedca44b1cba11a/clients/src/main/java/org/apache/kafka/clients/NodeApiVersions.java#L130 > > https://github.com/apache/kafka/blob/525b9b1d7682ae2a527ceca83fedca44b1cba11a/core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala#L195 > > Even if we did, we shouldn't return network exception errors. > > Do you mind opening a JIRA ticket with some more details so I can take a > closer look? > > Justine > > On Tue, May 7, 2024 at 11:38 AM Johnson Okorie <jokori...@gmail.com> > wrote: > > > Hi folks, > > > > Awesome work you have been doing on this project! > > > > I was hoping I could get some help on an issue we are having in one of > our > > Kafka clusters. Most of the clients on this cluster use > > exactly-once-semantics. The Kafka cluster currently runs version 3.5.2 > and > > we were attempting an upgrade to 3.6.2. After replacing one of the > brokers > > with the new version we saw a bunch of the following errors on the older > > brokers: > > > > ``` > > Received request api key ADD_PARTITIONS_TO_TXN with version 4 which is > not > > enabled > > ``` > > > > This manifested as 'NETWORK_EXCEPTION' errors on the clients and downtime > > for those clients. On the new broker we saw: > > > > ``` > > [AddPartitionsToTxnSenderThread-1063]: AddPartitionsToTxnRequest failed > for > > node 1069 with a network exception. > > ``` > > > > Digging through the changes in 3.6, we came across some changes > introduced > > as part of KAFKA-14402 < > https://issues.apache.org/jira/browse/KAFKA-14402> > > that > > we thought might lead to this behaviour and wanted to confirm. > > > > First we could see that transaction.partition.verification.enable > > is enabled by default and enables a new code path that culminates in we > > sending version 4 ADD_PARTITIONS_TO_TXN requests to other brokers here > > < > > > https://github.com/apache/kafka/blob/cb35ddc5ca233d5cca6f51c1c41b952a7e9fe1a0/core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala#L269 > > > > > . > > > > However, we do not support version 4 of ADD_PARTITIONS_TO_TXN requests > as > > of Kafka 3.5.2? If these assumptions happen to be correct, does this mean > > that the upgrade to versions 3.6+ require > > transaction.partition.verification.enable > > to be set to false to allow upgrades? > > > > Regard, > > Johnson > > >
