ok: - I start a Flink 1.17.1 cluster, run the job, then run `flink stop` and generate a savepoint. This savepoint will have Kryo 2.x data from standard Flink 1.17.1. - I start a Flink 1.18-SNAPSHOT cluster with the pull-request, run the job with resume from the savepoint from Flink 1.17, then I kill the cluster. I have a checkpoint with metadata. I believe this checkpoint is all using Kryo 5.x serialization. - I restart the cluster, run the job resuming from the checkpoint, and everything runs successfully. The job picks up where it left off and there are no errors, all output data looks correct.
Am I following the scenario correctly? Why would a checkpoint created by the new pull-request code have Kryo 2.x serialized data? Here is the code for my test app that I'm using. The checkpoint configuration settings are mostly from https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/fault-tolerance/checkpointing/ https://github.com/kurtostfeld/flink-kryo-upgrade-demo/blob/main/flink-kryo-upgraded/src/main/java/demo/app/Main.java ------- Original Message ------- On Thursday, June 8th, 2023 at 9:33 AM, Chesnay Schepler <ches...@apache.org> wrote: > > > On 08/06/2023 16:06, Kurt Ostfeld wrote: > > > If I understand correctly, the scenario is resuming from multiple > > checkpoint files or from a savepoint and checkpoint files which may be > > generated by different versions of Flink > > > No; it's the same version of Flink, you just didn't do a full migration > of the savepoint from the start. > > So, load old savepoint -> create an incremental checkpoint (which writes > > bit new state with Kryo5) -> jobs fails -> try recover job (which now > > has to read state was written with either Kryo2 or Kryo5). > > On 08/06/2023 16:06, Kurt Ostfeld wrote: > > > This pull-request build supports Java records > > > We'd have to see but of the top of my head I doubt we want to use Kryo > for that, and rather extend our PojoSerializer. At least so far that was > the plan.
