Re: kafka local single partition, what's the limit?

2018-08-22 Thread Ken Chen
You are reaching 10gb * 1000 / 64 = 156 MB / s which probably saturated your hard drive bandwidth ? So you can take a look at your iostats -- Sent from my iPhone On Aug 22, 2018, at 8:20 PM, Nan Xu wrote: I setup a local single node test. producer and broker are sitting at the same VM. broker

Re: Frequent consumer rebalances, auto commit failures

2018-06-01 Thread Ken Chen
1. Any detail logs ? 2. How do you process the records after you polled the records? 3. How much time does it take for every round of poll ? Thanks ! -- Sent from my iPhone On May 28, 2018, at 10:44 PM, Shantanu Deshmukh wrote: Can anyone here help me please? I am at my wit's end. I now have

Re: Kafka Connect task re-balance repeatedly

2018-03-22 Thread Ken Chen
Thanks Ewen. Will take a look at the config and if there are any findings, will come back here later. -- Sent from my iPhone On Mar 22, 2018, at 7:39 PM, Ewen Cheslack-Postava wrote: The log is showing that the Connect worker is trying to make sure it has read the entire log and gets to offset