Re: Is there an equivalent of a "synchronous subscribe"?

2022-01-25 Thread Richard Bosch
Hi Sebastian, I've encountered this as well during integration tests and have taken a different approach. When I need to verify the contents of topics with a single consumer I use the assign method. You do need to know the number of partitions of the topic as the argument used is a collection of T

Upgrade from 2.0 to 2.8.1 failed

2022-01-25 Thread Nicolas Carlot
Hello everyone, I just had a major failure while upgrading a kafka cluster from 2.0 to 2.8.1 following the provided migration process. I understand that a topicId is now given to each topic within zookeeper and meta.properties of each partition. While describing the topic, it seems I have differen

Re: Upgrade from 2.0 to 2.8.1 failed

2022-01-25 Thread Manoj.Agrawal2
Are u getting error at kafka log or server.log ? From: Nicolas Carlot Sent: Tuesday, January 25, 2022 6:20 AM To: users@kafka.apache.org Subject: Upgrade from 2.0 to 2.8.1 failed [External] Hello everyone, I just had a major failure while upgrading a kafka clu

Re: Upgrade from 2.0 to 2.8.1 failed

2022-01-25 Thread Nicolas Carlot
After a while all nodes seemed stable, but the producer errors because of min isr not attained. I had to restore our VM and didn't keep the logs... stupid me. What i'm really confused is the topicIds in zk here Le mar. 25 janv. 2022 à 15:13, a écrit : > Are u getting error at kafka log or server

Re: Upgrade from 2.0 to 2.8.1 failed

2022-01-25 Thread Israel Ekpo
Hi Nicolas, Did you follow the upgrade steps here? https://kafka.apache.org/documentation/#upgrade_2_8_1 Also, the recommended Zookeeper versions for 2.0 (3.4.13) is different from that of 2.8 (3.5.9) https://github.com/apache/kafka/blob/2.0/gradle/dependencies.gradle#L87 https://github.com/apa

Re: [ANNOUNCE] Apache Kafka 3.1.0

2022-01-25 Thread Israel Ekpo
Deepti, KIP-653 missed the cut for 3.1 so it was not part of this release https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.1.0 https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2 Israel Ekpo Lead Instructor, IzzyAcademy.com https://www.youtube.com

Re: [ANNOUNCE] Apache Kafka 3.1.0

2022-01-25 Thread Israel Ekpo
Bitfox, This is a great question, however it should be sent as a separate query to the mailing list instead of hijacking the release announcement. Currently, it might be possible to use these 4 things together but it is just a coincidence that they are all with similar version numbers. They versi

Re: Upgrade from 2.0 to 2.8.1 failed

2022-01-25 Thread Nicolas Carlot
Hi Israel, I followed these steps but instead of setting inter.broker.protocol.version to 2.8 in step 3 I just removed it. Dunno if that changes anything ? Regarding zookeeper we're using 3.5.8 Luckily this was a pre-production cluster, we were testing the upgrade here and we do have backups. Any

Re: Upgrade from 2.0 to 2.8.1 failed

2022-01-25 Thread Israel Ekpo
Hi Nicolas, Yes, those steps are important and very prescriptive. You need to follow those steps exactly as prescribed by the doctor :) In your scenario it needs to be set to 2.0 in the beginning and then once everything is stabilized and you are satisfied then you will have to bump it to 2.8 and

Re: Is there an equivalent of a "synchronous subscribe"?

2022-01-25 Thread Sebastian Proksch
Inspired by the `TestUtils` that Luke has referred to, I wrote a similar utility function and put together a minimal example [1] that IMHO exposes that the poll method behaves rather indeterministically or at least I cannot explain the observed behavior. In the example, I will always wait a to

Error triming topology

2022-01-25 Thread Murilo Tavares
Hi I have a KafkaStreams application that is too heavyweight, with 9 sub-topologies. I am trying to disable some unneeded part of the topology that is completely independent of the rest of the topology. Since my state stores have fixed, predictable names, I compared the topologies and I believe it