For changelog topics, their retention policy is based on the log compaction by default, and users can customize it if necessary when creating the state store (this is in trunk, not in released versions yet).
For repartition topics, their retention policy is based on whatever the broker's default retention policies. I suspect your main issue of relentless rebalancing is due to state stores being restoring when one or more instances are rolling bounced, which requires replaying the state store changelog topic from scratch. So if your changelog topic is very long, it will take very long time to restore. Guozhang On Thu, Dec 15, 2016 at 12:54 PM, Jon Yeargers <jon.yearg...@cedexis.com> wrote: > What's the retention settings for these (<topic>-changelog and > -replication)? Im wondering about the relentless rebalancing issues Im > facing and wondering if it has anything to do with consumers that lag too > far behind. > > If I delete all the topics associated with a KStream project and restart it > there are no rebalance issues. Everything is fast and responsive. > > Over the course of 6-10 hours of execution the rebalances take longer and > longer until eventually the app(s) stop responding at all. > > Just curious. > -- -- Guozhang
