Hi - we are building a stateful Flink streaming job that will run indefinitely. One part of the job builds up state per key in a global window that will need to exist for a very long time. We will definitely be using the savepoints to restore job state after new code deploys.
We were planning to be able to increase the parallelism of the job incrementally over time, as the volume of input data grows. We also have a large amount of historical data loaded into Kafka we'd like to reprocess initially with the streaming job to backfill Elasticsearch, and then transition the job seamlessly to nearline processing. We were planning to use a large parallelism during the historical reprocessing, and then decrease it when the job has caught up to new events. However, the savepoint docs state that the job parallelism cannot be changed over time [1]. Does this mean we need to use the same, fixed parallelism=n during reprocessing and going forward? Are there any tricks or workarounds we could use to still make changes to parallelism and take advantage of savepoints? Thanks, Zach [1] https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/savepoints.html#current-limitations
