Scaling Apache Kafka Producers & Consumers

2020-03-26 Thread Prasad Suhas Shembekar
Hi, I am using Apache Kafka as a Message Broker in our application. The producers and consumers are running as Docker containers in Kubernetes. Right now, the producer publishes messages to a topic in single partition. While the consumer consumes it from the topic. As per my understanding, in Ap

Re: Scaling Apache Kafka Producers & Consumers

2020-03-26 Thread M. Manna
Hey Prasad (#StayAtHomeSaveLives), On Thu, 26 Mar 2020 at 11:19, Prasad Suhas Shembekar < ps00516...@techmahindra.com> wrote: > Hi, > > I am using Apache Kafka as a Message Broker in our application. The > producers and consumers are running as Docker containers in Kubernetes. > Right now, the pr

Re: Scaling Apache Kafka Producers & Consumers

2020-03-26 Thread sunil chaudhari
Hi Prasad, Want to correct a bit. Ots not one consumer per partitions. Its one consumer thread per partitions. On Thu, 26 Mar 2020 at 4:49 PM, Prasad Suhas Shembekar < ps00516...@techmahindra.com> wrote: > Hi, > > I am using Apache Kafka as a Message Broker in our application. The > producers an

Re: MirrorMaker2 not mirroring for 5 minutes when adding a topic

2020-03-26 Thread Péter Sinóros-Szabó
So it seems that MM2 is doing a lot of small steps to get the offsets and consumer groups, am I right? Maybe this need some optimization or batching. Well I am just guessing here, I don't really know much about how it works. Do you have any idea maybe it if can be speed up with configuration? Tha

Re: Scaling Apache Kafka Producers & Consumers

2020-03-26 Thread Hans Jespersen
> As per my understanding, in Apache Kafka a single consumer from a consumer > group can consume messages from one partition only. Not correct. A single consumer from a consumer group can consume from many partitions. For example if you had a topic with 12 partitions and 4 consumers in a consume

Re: Scaling Apache Kafka Producers & Consumers

2020-03-26 Thread sunil chaudhari
Again A consumer can have one or more consumer thread. The analogy of 12 partitions and 4 consumer is true when each consumer has 3 consumer threads. Please don’t skip the important factor “consumer thread” in this matter. If you run each consumer with threads then you may need max 3 consumers

Re: leadership election timing on kafka broker failure

2020-03-26 Thread Larry Hemenway
I've tried to read up more on this issue and look at my logs. Here is what I think should is happening when we restart the controlling broker that also happens to be the leader of the partition in question: 1. Broker 0, the controlling broker that owns the partition we're looking at 2. The session

How to remove messed up segment log?

2020-03-26 Thread Richard Rossel
I was investigating a big topic, trying to find the reason why the retention limits were not working. I was able to read messages from 2019 even though the retention was for 5 days. Until I found the log segment file that contains those 2019 messages, and checking the rest of the messages I realize

Reg : Slowness in Kafka

2020-03-26 Thread Vidhya Sakar
Hi Team, The Kafka consumer is reading only 8 records per second.We have implemented apache Kafka and confluent connect S3. The confluent connect S3 collects the records and pushes it to S3 bucket. In this process, we are seeing some slowness like on an average only 8 records is being processed fo

Re: Reg : Slowness in Kafka

2020-03-26 Thread Hans Jespersen
Yes it should be going much faster than that. Something is wrong in your setup. -hans > On Mar 26, 2020, at 5:58 PM, Vidhya Sakar wrote: > > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have implemented > apache Kafka and confluent connect S3. The confluent connect

Re: Max poll interval and timeouts

2020-03-26 Thread Matthias J. Sax
`max.poll.intervall.ms` is the maximum allowed time between two calls to `poll()`. Hence, this config seems to be unrelated. For the background heartbeat thread there would be `session.timeout.ms` config but this also seems to be unrelated. What I don't fully understand is, what you try to achiev

Re: Get after put in stateStore returns null

2020-03-26 Thread Matthias J. Sax
Your code looks correct to me. If you write into the store, you should also be able to read it back from the store. Can you reproduce the issue using `TopologyTestDriver`? How many partitions does your input topic have? Is your stream partitioned by key? Note that `transfrom()` does not do auto-re

Re: Reg : Slowness in Kafka

2020-03-26 Thread Chris Larsen
Hi Vidhya, How many tasks are you running against the topic? How many partitions are on the topic? Can you post the connector config anonymized? Best, Chris On Thu, Mar 26, 2020 at 17:58 Vidhya Sakar wrote: > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have impleme

Re: Reg : Slowness in Kafka

2020-03-26 Thread James Olsen
Also check your Kafka Client and Server versions. There are serious latency issues when mixing different client and server versions IF your consumers handle multiple partitions. > On 27/03/2020, at 12:59, Chris Larsen wrote: > > Hi Vidhya, > > How many tasks are you running against the topic

Re: Scaling Apache Kafka Producers & Consumers

2020-03-26 Thread Hans Jespersen
Very good description with pictures in the book Kafka: The Definitive Guide https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html -hans > On Mar 26, 2020, at 12:00 PM, sunil chaudhari > wrote: > > Again > A consumer can have one or more consumer thread. > The