Re: Kafka stream application load balancing

2018-08-05 Thread Siva Ram
Hi Guozhang, Thanks for the suggestions below. I consider we got past the REBALANCING issue. However, we are running into significant memory usage issue. I will open a separate thread for this. 1) During the punctuate we require to perform certain tasks and it was exceeding the consumer reques

Re: Kafka stream application load balancing

2018-07-30 Thread Guozhang Wang
Hello Siva, To better understand your situation, I'd need to ask a few more questions: 1) What triggers your REBALANCING event? 2) Does your application contain any states? If yes, how are they configured (persistent or in-memory, is logging enabled, etc)? 3) What is your commit interval config

Kafka stream application load balancing

2018-07-29 Thread Siva Ram
Hi, Kafka version 1.0.0 (can't upgrade to another version yet due to legacy dependency) The stream application uses low level processor API and maintains state. A topic is setup with 30 partitions and I had split to 2 stream application instances consuming the same topic, each with 15 threads.