Hello Vijay, > Once i do that my flink consumer need to be restarted with changed parallelism. Why is this? The Flink consumer continuously scans for new shards, and will auto scale up/down the number of shard consumer threads to accommodate Kinesis resharding. Flink job/operator parallelism does not need to match the number of Kinesis shards.
> How do i solve this problem of restarting from existing checkpoint which was created with respect to N shards but now while restating we have more or less shards? The consumer should handle this. When the source starts it discovers active shards on the stream and attaches available state on a per shard basis. What is the problem here, is there an Exception you can share? Thanks, On Wed, Feb 23, 2022 at 11:25 AM Vijayendra Yadav <contact....@gmail.com> wrote: > Hi Team, > > I am running flink 1.11 kinesis consumer with say N kinesis shards, but i > want to increase/decrease shards to N+M or N-M. > Once i do that my flink consumer need to be restarted with changed > parallelism. > But i am unable to restart from existing checkpoint because of change in > number of shards. > > How do i solve this problem of restarting from existing checkpoint which > was created with respect to N shards but now while restating we have more > or less shards? > > Thanks, > Vijay
