To expand on Till's answer, Flink uses Kryo whenever the type system does not recognize a given type. If your job works with Kryo fallback disabled, there should be no usage of Kryo serialization in your snapshots[1]. In theory, it should be possible to upgrade Kryo using Flink's schema migration capabilities but I'm not aware of anyone who has looked into the details or is working on this.
Seth [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/types_serialization.html#disabling-kryo-fallback On Fri, Jan 15, 2021 at 10:46 AM Till Rohrmann <trohrm...@apache.org> wrote: > Yes, I think that Flink uses whatever key-/value-serializer is defined for > serializing the keys/values for a savepoint. > > I haven't looked in detail into the problem but if Kryo offers some > migration options, then this could make the problem a bit easier. > > Cheers, > Till > > > > On Fri, Jan 15, 2021 at 2:44 PM Niels Basjes <ni...@basjes.nl> wrote: > > > Till, > > > > So, we save the savepoints using the Kryo native format? > > But I see that Kryo 2.24 (as mentioned here) is from may 2014 ... > > that's ... really old ! > > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0 > > > > I see that Kryo has some suggestions on the migration options: > > > > > https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide > > One thing is that they have an extra artifact that allows having multiple > > versions of Kryo in a single application to make this possible. > > > > Niels Basjes > > > > On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <trohrm...@apache.org> > > wrote: > > > > > Hi Fred, > > > > > > one reason why this hasn't been updated is that Flink needs to ensure > > > backwards compatibility. Hence, if the newer Kryo library serializes > > > objects differently, then Flink would need to migrate old savepoints > > having > > > been stored via the old Kryo serializer to the new format. This is > > > currently not supported and afaik blocks upgrading this library. > > > > > > Cheers, > > > Till > > > > > > On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred) > > > <fred.teunis...@ing.com.invalid> wrote: > > > > > > > Hi, > > > > > > > > While I was resolving a dependency version conflict in our project > that > > > > uses Flink v1.11.2 I noticed that Flink is using the library > > > > ‘com.esotericsoftware.kryo:kryo< > > > > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo > > > > >:2.24.0’. > > > > Is there a reason why this library isn’t updated to a more recent > > version > > > > (the last one is `com.esotericsoftware: kryo< > > > > https://mvnrepository.com/artifact/com.esotericsoftware/kryo > >:5.0.3`) > > > > > > > > Note. The groupId has been changed, so that could be the reason why > it > > is > > > > missed. > > > > > > > > Regards, > > > > Fred Teunissen > > > > > > > > ----------------------------------------------------------------- > > > > ATTENTION: > > > > The information in this e-mail is confidential and only meant for the > > > > intended recipient. If you are not the intended recipient, don't use > or > > > > disclose it in any way. Please let the sender know and delete the > > message > > > > immediately. > > > > ----------------------------------------------------------------- > > > > > > > > > > > > > -- > > Best regards / Met vriendelijke groeten, > > > > Niels Basjes > > >
