Re: Yet Another Repartitioning Question About Kafka Streams

2021-04-14 Thread Guozhang Wang
Hello Gareth, There is a checkpoint file that records the corresponding offset of the changelog for the state store data co-located within the state directory; after the partition is migrated to new owners, this checkpoint file along with the state store would not be deleted immediately but follow

Re: Yet Another Repartitioning Question About Kafka Streams

2021-04-11 Thread Gareth Collins
Hi Guozheng, Thanks very much again for the answers! One follow-up on the first question. Just so I understand it, how would it know where to continue from? I would assume that once we repartition, the new node will own the position in the consumer group for the relevant partition(s) so Kafka/Zoo

Re: Yet Another Repartitioning Question About Kafka Streams

2021-04-05 Thread Guozhang Wang
Hello Gareth, 1) For this scenario, its state should be reusable and we do not need to read from scratch from Kafka to rebuild. 2) "Warmup replicas" is just a special standby replica that is temporary, note that if there's no partition migration needed at the moment, the num.warmup.replicas is ac

Yet Another Repartitioning Question About Kafka Streams

2021-04-04 Thread Gareth Collins
Hi, Thanks very much for answers to my previous questions here. I had a couple more questions about repartitioning and I just want to confirm my understanding. (1) Given the following scenario: (a) I have a cluster of Kafka stream nodes with partitions assigned to each. (b) One node goes down.