Re: duplicate packets in kafka topic

Hi, What are duplicate messages in your use case? 1) different messages with the same content 2) the same message that is send multiple times to the broker due to retries in the producer 3) something else What do you mean with "identify those duplicates"? What do you want to do with them? For ca

Segments Not Compacted Due to Empty Transactional Record Batches

We're seeing an issue with some __consumer_offsets partitions that host consumer groups being used in EOS-style read-process-write cycles. Although the brokers' log cleaners are running, we are finding more and more segment files for these partitions, for example: ``` $ ls -lh data/__consumer_offs

duplicate packets in kafka topic

I have multiple clients who can send duplicate packets multiple time to same kafka topic.. Is there a way to identify those duplicate packets.

Re: Required guidelines for kafka upgrade

Answers inline. first what we upgrade zookeeper or kafka?? - I can't really comment about ZK upgrade. Recently i upgraded only 10 node Kafka Cluster in production. Some of the Key Notes : 1) You need to set inter.broker.protocol.version = 1.1.0( Your Current Broker Version ) 2) Start upg

Re: Required guidelines for kafka upgrade

Ashok, The guideline is well-explained, so please follow that. https://kafka.apache.org/documentation/#upgrade The process works, so try and follow what it recommends. Thanks, On Fri, 26 Apr 2019 at 12:36, ASHOK MACHERLA wrote: > Dear senthil > > That I know, > Could you please explain in de

Mirro maker tool is not working

Dear Team Right now, Kafka 0.10.1, Zookeeper is 3.4.6 version are using in our production cluster. In which we are using mirror maker tool to transfer data from DC cluster to DR cluster. We have 3 brokers and 3 zookeepers are using in both environments. In DC , server.properties we set message

Re: Required guidelines for kafka upgrade

Dear senthil That I know, Could you please explain in details, any documents, step by steps required. first what we upgrade zookeeper or kafka?? being upgrade can we pointing old logs directory?? Could please explain like that. Sent from Outlook ___

Re: Required guidelines for kafka upgrade

Hi , You can refer official documentation to upgrade Kafka Cluster . Section: 1.5 Upgrading From Previous Versions Last week we did broker upgrade from 1.1.0 to 2.2.0. I think the current stable version is 2.2.0. --Senthil On Fri, Apr 26, 2019, 3:54 PM ASHOK MACHERLA wrote: > Dear Team > > Rig

Required guidelines for kafka upgrade

Dear Team Right now, we are using Kafka 0.10.1 version, zookeeper 3.4.6 version, we are planning to upgrade to New Kafka & Zookeeper version. Please suggest us, what is current stable version of Kafka, Zookeeper??? Could you please explain what is steps to upgrade to Kafka, Zookeeper?? Is ther