Long log appending time

Hi, all I'm trying to optimize the long tail latency in kafka. My OS is centos 4.3 kernel 2.6.32. Java version 1.8.0_45 kafka version 0.10.2. The disk is sata HDD. My log: [2017-12-22 15:29:58,156] TRACE [ReplicaFetcherThread-2-1], Follower 2 has replica log end offset 10428 for partition kafka-

Re: Long log appending time

by the way, I have 3 brokers, the producing rate is about 250 messages/sec.. 2017-12-22 16:20 GMT+08:00 陈江枫 : > Hi, all > > I'm trying to optimize the long tail latency in kafka. > My OS is centos 4.3 kernel 2.6.32. Java version 1.8.0_45 > kafka version 0.10.2. > The disk is sata HDD. > > My log

Kafka Stream - How to calculate and send a Diff message in an aggregation method

Hello, I'm using KafkaStream for some months and I don't succeed to implement this use case : I need to compare two objects : a new one and the old one. At the end, I need to send a message with the diff between the 2 objets. My first implementation was to use an aggregate and to return the diff

Re: Kafka Stream - How to calculate and send a Diff message in an aggregation method

Hi, what I don't understand is what you mean by >> But when I am reading the KTable, I have no guarantee to see all messages >> with the same key (because of the commit.interval.ms configuration). Can you elaborate? I actually think an aggregation should be the correct operator to use. However,

Re: Kafka Streams 0.11 consumers losing offsets for all group.ids

Thanks for reporting this. What is your `offsets.topic.replication.factor`? -Matthias On 12/19/17 8:32 AM, Adam Gurson wrote: > I am running two kafka 0.11 clusters. Cluster A has two 0.11.0.0 brokers > with 3 zookeepers. Cluster B has 4 0.11.0.1 brokers with 5 zookeepers. > > We have recen

Re: Kafka Streams - max.poll.interval.ms defaults to Integer.MAX_VALUE

The value was change to make Streams application robust against large state restore phases during rebalance. Ie, it is targeted to exactly "fix" 2. If an application needs to restore state, this state restore might take longer than the max.poll.interval.ms parameter and thus, even if the applicati

Re: Consumer group describe issue

Your observation is correct. KafkaConsumer.position() is a blocking call. It's a know issue that there is no configurable timeout value. I am not aware of any workaround. -Matthias On 12/21/17 6:05 AM, sahil aggarwal wrote: > Hi, > > Facing issue where *kafka-consumer-groups.sh --describe * ge

Re: Consumer group describe issue

Fixed it by some code change in ConsumerGroupCommand.scala. Possible to push it upstream for 0.10.* ? It seems to be fixed in 0.11.* where it uses getEndOffsets() which has requestTimeoutMs instead of Long.MAX_VALUE. On 23 December 2017 at 02:46, Matthias J. Sax wrote: > Your observation is cor

Re: Consumer group describe issue

Sahil: I did a quick search in 0.11.0 branch and trunk for getEndOffsets but didn't find any occurrence. Mind giving us the location (and class) where getEndOffsets is called ? Thanks On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal wrote: > Fixed it by some code change in ConsumerGroupCommand

Re: Consumer group describe issue

my bad its consumer.endOffsets in ConsumerGroupCommand.scala. https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea4f77ef99d4db15373/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L467 On 23 December 2017 at 13:07, Ted Yu wrote: > Sahil: > I did a quick search in 0.11.0 bran