Antony Stubbs created KAFKA-6145:
------------------------------------
Summary: Warm up new KS instances before migrating tasks -
potentially a two phase rebalance
Key: KAFKA-6145
URL: https://issues.apache.org/jira/browse/KAFKA-6145
Project: Kafka
Issue Type: New Feature
Components: streams
Reporter: Antony Stubbs
Currently when expanding the KS cluster, the new node's partitions will be
unavailable during the rebalance, which for large states can take a very long
time, or for small state stores even more than a few ms can be a deal breaker
for micro service use cases.
One workaround would be two execute the rebalance in two phases:
1) start running state store building on the new node
2) once the state store is fully populated on the new node, only then rebalance
the tasks - there will still be a rebalance pause, but would be greatly reduced
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
