Hi Mahendra, Just a kind reminder that upgrading Streams to 0.10.2 does not necessarily require you to upgrade brokers to 0.10.2 as well. Since we have added a new feature since 0.10.2 to allow newer versioned clients (producer, consumer, streams) to talk to older versioned brokers, and for Streams specifically it only requires brokers to be no older than 0.10.1.
Guozhang On Mon, Mar 13, 2017 at 5:12 AM, Mahendra Kariya <mahendra.kar...@go-jek.com > wrote: > We are planning to migrate to the newer version of Kafka. But that's a few > weeks away. > > We will try setting the socket config and see how it turns out. > > Thanks a lot for your response! > > > > On Mon, Mar 13, 2017 at 3:21 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > > > Thanks, > > > > A couple of things: > > - I’d recommend moving to 0.10.2 (latest release) if you can since > several > > improvements were made in the last two releases that make rebalancing and > > performance better. > > > > - When running on environments with large latency on AWS at least > (haven’t > > tried Google cloud), one parameter we have found useful to increase > > performance is the receive and send socket size for the consumer and > > producer in streams. We’d recommend setting them to 1MB like this (where > > “props” is your own properties object when you start streams): > > > > // the socket buffer needs to be large, especially when running in AWS > with > > // high latency. if running locally the default is fine. > > props.put(ProducerConfig.SEND_BUFFER_CONFIG, 1024 * 1024); > > props.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 1024 * 1024); > > > > Make sure the OS allows the larger socket size too. > > > > Thanks > > Eno > > > > > On Mar 13, 2017, at 9:21 AM, Mahendra Kariya < > mahendra.kar...@go-jek.com> > > wrote: > > > > > > Hi Eno, > > > > > > Please find my answers inline. > > > > > > > > > We are in the process of documenting capacity planning for streams, > stay > > tuned. > > > > > > This would be great! Looking forward to it. > > > > > > Could you send some more info on your problem? What Kafka version are > > you using? > > > > > > We are using Kafka 0.10.0.0. > > > > > > Are the VMs on the same or different hosts? > > > > > > The VMs are on Google Cloud. Two of them are in asia-east1-a and one is > > in asia-east1-c. All three are n1-standard-4 Ubuntu instances. > > > > > > Also what exactly do you mean by “the lag keeps fluctuating”, what > > metric are you looking at? > > > > > > We are looking at Kafka Manager for the time being. By fluctuating, I > > mean the lag is few thousands at one time, we refresh it the next second, > > it is in few lakhs, and again refresh it and it is few thousands. I > > understand this may not be very accurate. We will soon have more accurate > > data once we start pushing the consumer lag metric to Datadog. > > > > > > But on a separate note, the difference between lags on different > > partitions is way too high. I have attached a tab separated file herewith > > which shows the consumer lag (from Kafka Manager) for the first the 50 > > partitions. As is clear, the lag on partition 2 is 530 while the lag on > > partition 18 is 23K. Note that the same VM is pulling data from both the > > partitions. > > > > > > > > > > > > > > > <KafkaLags.tsv> > > > > > -- -- Guozhang
