Re: Managed State Custom Serializer with Avro

Hi guys, Noticing similar things as Arvid mentions. I currently solved the issue by also supporting GenericRecords written and read with the old schema and parse them to the new schema myself. This at least gives us the evolution until state migration is there. Thanks for your help! Cheers, Nie

Re: Managed State Custom Serializer with Avro

Hi Aljoscha, I opened https://issues.apache.org/jira/browse/FLINK-8715 for the RocksDB issue with pointers to the code. Let me know if you need more details. Best, Arvid On Tue, Feb 20, 2018 at 1:04 PM, Arvid Heise wrote: > Hi Aljoscha, hi Till, > > @Aljoscha, the new AvroSerializer is almost

Re: Managed State Custom Serializer with Avro

Hi Aljoscha, hi Till, @Aljoscha, the new AvroSerializer is almost what I wanted except that it does not use the schema of the snapshot while reading. In fact, this version will fail with the same error as before when a field is added or removed. https://github.com/apache/flink/blob/f3a2197a2352404

Re: Managed State Custom Serializer with Avro

A small addition, currently savepoints are always full checkpoints. Thus, you should not have the problem when calling cancel with savepoint. Concerning 2, I think the idea was to only check for compatibility at restore time. The check will either say its compatible or not. If it's not compatible

Re: Managed State Custom Serializer with Avro

Hi Arvid, Did you check out the most recent AvroSerializer code? https://github.com/apache/flink/blob/f3a2197a23524048200ae2b4712d6ed833208124/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java#L185

Re: Managed State Custom Serializer with Avro

Hi guys, just wanted to write about that topic on my own. The FF talk of Tzu-Li gave me also the impression that by just using AvroSerializer, we get some kind of state evolution for free. https://www.slideshare.net/FlinkForward/flink-forward-berlin-2017-tzuli-gordon-tai-managing-state-in-apache-

Re: Managed State Custom Serializer with Avro

Hi Till, Thanks for the quick reply, I'm using 1.3.2 atm. Cheers, Niels On Feb 19, 2018 19:10, "Till Rohrmann" wrote: > Hi Niels, > > which version of Flink are you using? Currently, Flink does not support to > upgrade the TypeSerializer itself, if I'm not mistaken. As you've > described, it w

Re: Managed State Custom Serializer with Avro

Hi Niels, which version of Flink are you using? Currently, Flink does not support to upgrade the TypeSerializer itself, if I'm not mistaken. As you've described, it will try to use the old serializer stored in the checkpoint stream to restore state. I've pulled Gordon into the conversation who ca