Re: Yet Another Repartitioning Question About Kafka Streams

2021-04-14 Thread Guozhang Wang
Hello Gareth, There is a checkpoint file that records the corresponding offset of the changelog for the state store data co-located within the state directory; after the partition is migrated to new owners, this checkpoint file along with the state store would not be deleted immediately but follow

Re: Yet Another Repartitioning Question About Kafka Streams

2021-04-11 Thread Gareth Collins
Hi Guozheng, Thanks very much again for the answers! One follow-up on the first question. Just so I understand it, how would it know where to continue from? I would assume that once we repartition, the new node will own the position in the consumer group for the relevant partition(s) so Kafka/Zoo

Re: Yet Another Repartitioning Question About Kafka Streams

2021-04-05 Thread Guozhang Wang
Hello Gareth, 1) For this scenario, its state should be reusable and we do not need to read from scratch from Kafka to rebuild. 2) "Warmup replicas" is just a special standby replica that is temporary, note that if there's no partition migration needed at the moment, the num.warmup.replicas is ac