Re: Avro Schema Resolution Compatibility

2019-11-04 Thread Arvid Heise
Hi Dominik, just to add to Dawids explanation: to have a proper schema evolution on Avro data, it needs to know the schema with which it was written. For state that means that we are storing the used schema once for all state records in the state file, since they all belong to the same schema vers

Re: Avro Schema Resolution Compatibility

2019-11-04 Thread Dominik Wosiński
Hey Dawid, Thanks a lot. I have indeed missed the part that this is actually about State not the Deserialization itself. This seems to be clear and consistent now. Thanks again, Best Regards, Dom. pon., 4 lis 2019 o 13:18 Dawid Wysakowicz napisał(a): > Hi Dominik, > > I am not sure which docum

Re: Avro Schema Resolution Compatibility

2019-11-04 Thread Dawid Wysakowicz
Hi Dominik, I am not sure which documentation do you refer to when saying: "According to the docs the schema resolution is compatible with the Avro docs", but I assume this one[1]. If this is the case then the AvroDeserializationSchema plays no role in this process. That page describes evolution o

Avro Schema Resolution Compatibility

2019-11-04 Thread Dominik Wosiński
Hey, I have a question regarding Avro Types and schema evolution. According to the docs the schema resolution is compatible with the Avro docs [1]. But I have done some testing. For example, I have created a record, written it to Kafka, and then changed the order the fields in schema and tried to