Hi Lars, That sounds like a painful process. Since the offsets are inconsistent, I would suggest to reset the Kafka source state by changing the `uid`, set the source to start from earliest if you haven't already, make the bootstrap server change, and restart your job with allowNonRestoredState enabled. This process effectively will retain Flink state, excluding the Kafka source.
Here is a similar question, where Martjin gave the same answer, but about resetting Kafka topics: https://lists.apache.org/thread/xcfjm23xk7xy9nh887pvsxbw9z649p3q. If you are interested, I talked about this exact problem at Flink Forward and how we are trying to solve it with FLIP-246 https://cwiki.apache.org/confluence/display/FLINK/FLIP-246%3A+Multi+Cluster+Kafka+Source. The discussion thread is linked at the topic if you want to give feedback on the FLIP. Best, Mason On Thu, Oct 6, 2022 at 9:40 AM Lars Skjærven <[email protected]> wrote: > Hello, > > What is the recommended approach for migrating flink jobs to a new kafka > server? I was naively hoping to use Kafka Mirror Maker to sync the old > server with the new server, and simply continue from savepoint with updated > URL's. Unfortunately, the kafka offsets are not identical for log compacted > topics when using mirror maker. Any tips ? > > L >
