Sameer, the KIP you are pointing to is not related to Kafka Streams' task/partition assignment. Kafka Streams uses it's own implementation of a partitioning assigner (not the default one the consumer uses).
-Matthias On 1/9/18 4:22 AM, Sameer Kumar wrote: > Got It. Thanks. Others can also take a look at > https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy > > -Sameer. > > On Tue, Jan 9, 2018 at 5:33 PM, Damian Guy <damian....@gmail.com> wrote: > >> Hi, >> >> yes partition assignment is aware of the standby replicas. It will try and >> assign tasks to the nodes that have the state for the task, but also will >> try and keep the assignment balanced. >> So the assignment will be more like your second assignment. If you are >> interested you can have a look at: >> https://github.com/apache/kafka/blob/trunk/streams/src/ >> test/java/org/apache/kafka/streams/processor/internals/assignment/ >> StickyTaskAssignorTest.java >> >> >> On Tue, 9 Jan 2018 at 11:44 Sameer Kumar <sam.kum.w...@gmail.com> wrote: >> >>> Hi Damian, >>> >>> Thanks for your reply. I have some further ques. >>> >>> Would the partition assignment be aware of the standby replicas. What >> would >>> be the preference for task distribution: load balancing or stand by >>> replicas. >>> >>> For e.g >>> >>> N1 >>> assigned partitions: 1,2 >>> standby partitions: 5,6 >>> >>> N2 >>> assigned partitions: 3,4 >>> standby partitions: 1,2 >>> >>> N3 >>> assigned partitions: 5,6 >>> standby partitions: 3,4 >>> >>> After N1 goes down, what would be the state of the cluster >>> >>> N2 >>> assigned partitions: 3,4,1,2 >>> standby partitions: 5,6 >>> >>> N3 >>> assigned partitions: 5,6 >>> standby partitions: 3,4,1,2 >>> >>> Or >>> >>> N2 >>> assigned partitions: 3,4,1 >>> standby partitions: 2,5,6 >>> >>> N3 >>> assigned partitions: 5,6,2 >>> standby partitions: 1,3,4 >>> >>> -Sameer. >>> >>> On Tue, Jan 9, 2018 at 2:27 PM, Damian Guy <damian....@gmail.com> wrote: >>> >>>> On Tue, 9 Jan 2018 at 07:42 Sameer Kumar <sam.kum.w...@gmail.com> >> wrote: >>>> >>>>> Hi, >>>>> >>>>> I would like to understand how does rebalance affect state stores >>>>> migration. If I have a cluster of 3 nodes, and 1 goes down, the >>>> partitions >>>>> for node3 gets assigned to node1 and node2, does the rocksdb on >>>> node1/node2 >>>>> also starts updating its store from changelog topic. >>>>> >>>>> >>>> Yes the stores will be migrated to node1 and node2 and they will be >>>> restored from the changelog topic >>>> >>>> >>>>> If yes, then what impact would this migration process have on >> querying. >>>>> >>>> >>>> You can't query the stores until they have all been restored and the >>>> rebalance ends. >>>> >>>>> >>>>> Also, if the state store restoration process takes time, how to make >>> sure >>>>> another rebalance doesn''t happen. >>>>> >>>>> >>>> If you don't lose any more nodes then another rebalance won't happen. >> If >>>> node1 comes back online, then there will be another rebalance, however >>> the >>>> time taken shouldn't be as long as it will already have most of the >> state >>>> locally, so it only needs to catch up with the remainder of the >>> changelog. >>>> Additionally, you should run with standby tasks. They are updated in >> the >>>> background and will mean that in the event of failure the other nodes >>>> should already have most of the state locally, so the restoration >> process >>>> won't take so long >>>> >>>> >>>>> -Sameer. >>>>> >>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
