Re: Dead broker in ISR

How did you reassign partitions? While reassigning reassignment json had the broker 0 mentioned, it could be in this state. Could you share what's the output of describing the topic from console. On Wed, Feb 19, 2020 at 5:11 AM Bhat, Avinash wrote: > Hi Ivan, > > This is probably by design, sinc

Re: Kafka Cluster High Produce Time

ecific follower is causing the issue (if it's network) since we' ve brokers across different regions. Thanks, Dinesh Kumar On Thu, Aug 22, 2019 at 11:14 AM Lisheng Wang wrote: > Hi Dinesh > > Just wanna check if the metrics you called is "RemoteTimeMs" or not? >

Kafka Cluster High Produce Time

red are followers. 1. Is there any configuration we could tweak to reduce the produce time 2. What's the next step to say debug why remote produce time is high. Thanks, Dinesh Kumar

Re: Kafka Rebalance on Watcher event Question

consumer failure, there will be a re-balance to assign the > failed topic/partitions to live consumers. > > pl check consumer documentation here > https://kafka.apache.org/documentation.html#introduction > > On Mon, May 11, 2015 at 11:17 AM, dinesh kumar > wrote: > > >

Re: Kafka Rebalance on Watcher event Question

mer group, then the re-balance will be > triggered. > A consumer subscribes to event changes of the consumer id registry within > its group. > > On Mon, May 11, 2015 at 10:55 AM, dinesh kumar > wrote: > > > Hi, > > I am looking at the code of > kafka.consumer.Zoo

Kafka Rebalance on Watcher event Question

Hi, I am looking at the code of kafka.consumer.ZookeeperConsumerConnector.scala (link here ) and I see that all ids registered to a particular group ids are registered to the path /consu

Re: Poor performance consuming multiple topics

Hi CJ, I recently ran into some kafka message size related issue and did some digging around to understand the system. I will put those details in brief and hope it will help you. Each consumer connector has fetcher threads and fetcher manager threads associated with it. The Fetcher thread talks t

Re: Consumer not getting data when there is a big lag in the topic

We don't have any compression on Kafka On 14 January 2015 at 22:54, Guozhang Wang wrote: > Did you have compression enabled on Kafka? > > On Tue, Jan 13, 2015 at 10:33 AM, dinesh kumar > wrote: > > > We are using 0.8.1.1 version of Kafka and *not 0.8.2 *as menti

Re: Consumer not getting data when there is a big lag in the topic

We are using 0.8.1.1 version of Kafka and *not 0.8.2 *as mentioned above. Thanks, Dinesh On 13 January 2015 at 23:35, dinesh kumar wrote: > Hi Guozhang, > Sorry for the misinformation. We have file sizes around 50 - 100 MB. So we > set *fetch.message.max.bytes* conservatively around

Re: Consumer not getting data when there is a big lag in the topic

> Your fetch.message.max.bytes is 188743680 < 155MB, but you said some > messages can be as large as 180MB. Could you try to set it to be larger > than, say 200MB and see if it helps? > > Guozhang > > On Tue, Jan 13, 2015 at 4:18 AM, dinesh kumar wrote: > > > Hi, &g

Consumer not getting data when there is a big lag in the topic

Hi, I am been facing some JAVA high level consumer related issues lately and would like to understand more on this. We have 9 bare-metals (48 core, 250 GB, Terabytes of Hard disks) running *Kafka 0.8.2* and 5 independent VM (8 core, 60 GB) running zookeeper. I have a topic that has key as metadat

Consumer reading from same topic twice without commiting

Hi, I am trying to understand the behavior of Kafka Consumer in Java. Consider a scenario where a consumer is reading from a partition with auto.comiit disabled and only one partition in the topic. The consumer reads a message 'A' and sleep for some time, say 10 seconds. Then reads from the same p

Kafka Release timelines

Hi, I found the future release plan wiki here . I see that 0.8.2 is still in beta even though it was stated for September. What is the expected date for 0.9 release? Thanks, Dinesh

Re: How costly is Re balancing of partitions for a topic

or all the consumers to commit offsets will it wait for 5 minutes? Or is there a timeout here? If the consumer does not commit after 5 minutes due to some exception what will happen? Thanks, Dinesh On Wed, Nov 5, 2014 at 10:22 PM, dinesh kumar wrote: > Thanks for the answers. Have some follow

Re: How costly is Re balancing of partitions for a topic

umer client, which will reduce > rebalance latency compared to the current consumer. > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design > > > Guozhang > > > > > > > On Wed, Nov 5, 2014 at 4:50 AM, dinesh kumar wrote: > > > H

How costly is Re balancing of partitions for a topic

Hello, I am trying to come up with a design for consuming from Kafka. *I am using 0.8.1.1 version of Kafka. *I am thinking of designing a system where the consumer will be created every few seconds, consume the data from Kafka, process it and then quits after committing the offsets to Kafka. At a