Re: restart Kafka Streams application takes around 5 minutes

2017-03-16 Thread Tianji Li
Hi Guozhang, I am using 0.10.2.0. Will try the trunk and report back. Thanks Tianji On Wed, Mar 15, 2017 at 5:25 PM, Guozhang Wang wrote: > Are you building with any released versions of Kafka or with a build from > Kafka trunk? There are a few fixes we have made post 0.10.2 in trunk that > ha

Re: restart Kafka Streams application takes around 5 minutes

2017-03-15 Thread Guozhang Wang
Are you building with any released versions of Kafka or with a build from Kafka trunk? There are a few fixes we have made post 0.10.2 in trunk that has largely reduced the rebalance latency so I'd recommend try using a build from Kafka trunk for testing if possible. Guozhang On Wed, Mar 15, 2017

Re: restart Kafka Streams application takes around 5 minutes

2017-03-15 Thread Tianji Li
It seems independent to the rocksdb sizes. It also took 5 minutes when there are 375Mbytes this morning... On Wed, Mar 15, 2017 at 9:13 AM, Sachin Mittal wrote: > rocksdb state store initialization may be taking up that time. > Whats the size of your rockksdb state directory. May be partitioning

Re: restart Kafka Streams application takes around 5 minutes

2017-03-15 Thread Sachin Mittal
rocksdb state store initialization may be taking up that time. Whats the size of your rockksdb state directory. May be partitioning the source topic, increasing the number of threads/instances processing the source and reducing the time window of aggregation can help in reducing the startup time.

restart Kafka Streams application takes around 5 minutes

2017-03-15 Thread Tianji Li
Hi there, In the experiments I am doing now, if I restart the streams application, I have to wait for around 5 minutes for some reason. I can see something in the Kafka logs: [2017-03-15 08:36:18,118] INFO [GroupCoordinator 0]: Preparing to restabilize group xxx-test25 with old generation 2 (kaf