Re: Savepoints and migrating value state data types

Actually, Flink 1.4 will come with improved Avro support. See especially: - https://issues.apache.org/jira/browse/FLINK-7420: Move All Avro Code to flink-avro - https://issues.apache.org/jira/browse/FLINK-7997:

Re: Savepoints and migrating value state data types

Hi Gordon I've been looking into creating a custom AvroSerializer without Kryo which would support Avro schemas and I'm starting to wonder if this is actually the most straightforward way to do it. If I extend a class from TypeSerializer I would also need to implement a TypeInformation class to

Re: Savepoints and migrating value state data types

Hi, Yes, the AvroSerializer currently partially still uses Kryo for object copying. Also, right now, I think the AvroSerializer is only used when the type is recognized as a POJO, and that `isForceAvroEnabled` is set on the job configuration. I’m not sure if that is always possible. As mentioned

Re: Savepoints and migrating value state data types

Gordon Thanks for the detailed response. I have verified your assumption and that is, unfortunately, the case. I also looked into creating a custom Kryo serializer but I got stuck on serializing arrays of complex types. It seems like this isn't trivial at all with Kryo. As an alternative, I've b

Re: Savepoints and migrating value state data types

Hi! The exception that you have bumped into indicates that on the restore of the savepoint, the serializer for that registered state in the savepoint no longer exists. This prevents restoring savepoints taken with memory state backends because there will be no serializer available to deserialize t

Re: Savepoints and migrating value state data types

Hi Marc, I assume you have set a UID for your CoProcessFunction as described in [1]? Also, can you provide the Flink version you are working with and the serializer you are using? If you have the UID set, your strategy seems to be the same as proposed by [2]: "Although it is not possible to chan