Sorry to keep prodding you with questions, but can you quantify the increase for the ProduceRequest? What is the workload you are testing against: specificallly the batch size, message size, linger time settings of the producers in question?
I ask because we benchmarked 0.11.0 against the older 0.10.0 message format and found no difference in performance between an 0.10.2 on the 0.10 message format and 0.11.0 on the 0.10 message format. Could you create a topic with the 0.10.0 message format and see if there is any degradation for the same workload? Thanks, Apurva On Tue, Aug 1, 2017 at 2:51 AM, Gaurav Abbi <abbi.gau...@gmail.com> wrote: > Hi Apurva, > There are increases in the *Produce* request also. It is not as substantial > as compared to *OffsetCommit. *For both of these requests, the major > contributor is Remote time. > A couple of other metrics that show different behavior post upgrade: > > 1. *LogStartOffset*: It has drastically decreased. > 2. *NumDelayedOperations: *It has dropped. > > These could be related or may be these are intended good changes in Kafka > 0.11.0.0 or one of the previous versions. > > Best Regards, > Gaurav Abbi > > On Tue, Aug 1, 2017 at 12:11 AM, Apurva Mehta <apu...@confluent.io> wrote: > > > Thanks for your response. Is it 200% only for the OffsetCommitRequest, or > > is it similar for all the requests? > > > > > > On Mon, Jul 31, 2017 at 12:48 PM, Gaurav Abbi <abbi.gau...@gmail.com> > > wrote: > > > > > Hi Apurva, > > > 1. The increase is about 200%. > > > 2. There is no increase in throughput. However, this has caused in > error > > > rate and a decrease in the responses received per second. > > > > > > > > > One more thing to mention, we also upgraded to 0.11.0.0 client > libraries. > > > We are currently using old Producer and consumer APIs. > > > > > > > > > > > > Best Regards, > > > Gaurav Abbi > > > > > > On Mon, Jul 31, 2017 at 7:46 PM, Apurva Mehta <apu...@confluent.io> > > wrote: > > > > > > > How much is the increase? Is there any increase in throughput? > > > > > > > > On Mon, Jul 31, 2017 at 8:04 AM, Gaurav Abbi <abbi.gau...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > We recently upgraded to Kafka 0.11.0.0 from 0.10.1.1. > > > > > Since then we have been observing increased latencies especially > > > > > OffsetCommit requests. > > > > > Looking at the server side metrics, it seems the culprit is the > > > Follower > > > > > time. > > > > > > > > > > We are using following > > > > > inter.broker.protocol.version: 0.11.0.0 > > > > > log.message.format.version: 0.9.0.1 > > > > > > > > > > Are there some possible pointers that we can explore to > troubleshoot > > > the > > > > > root cause? > > > > > > > > > > Best Regards, > > > > > Gaurav Abbi > > > > > > > > > > > > > > >
