Re: Key state does not support migration

2019-06-03 Thread Tzu-Li (Gordon) Tai
Hi Richard, Schema evolution for data types that are used as keys is not allowed because, potentially, if the schema of the key changes, hash codes of keys may also change and can break partitioning for internal state managed by Flink. There are of course some evolution scenarios that would not re

Re: Key state does not support migration

2019-06-03 Thread Till Rohrmann
Hi Richard, I've pulled in Gordon who worked on this feature. He should be able to tell you about the current limitations of Flink's schema evolution. Cheers, Till On Wed, May 29, 2019 at 1:44 PM Richard Deurwaarder wrote: > Hello, > > I am running into the problem where (avro) schema evolutio

Key state does not support migration

2019-05-29 Thread Richard Deurwaarder
Hello, I am running into the problem where (avro) schema evolution works perfectly for operator/keyed state but does not work when used with keyBy(). For example: I have a job like so: env.addSource(someSource()) .keyBy(object -> getMyAvroObject()) .process(doSomething()