Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4943
Thanks for the review, addressing the comments.
I found the problem with the backwards compatibility: It is the Avro
version upgrade.
Avro types generated with Avro 1.7.7 (Flink 1.3) and serialized as PoJos
don't work when read into PoJos generated from the same Schema in Avro 1.8.2
(Flink 1.4). The Avro class structure changed, breaking the Pojo compatibility.
Avro is binary format compatible, but not class compatible across versions,
it seems. Makes sense, for a serialization format.
Now, this brings up a few issues:
- Users that use generated Avro types from earlier Avro versions would be
forced to upgrade (re-create) these classes with the new Avro version, if they
upgrade Flink and it pulls in a newer Avro version. We might solve that by
removing avro from `flink-dist` and pushing it into "user code space", where
users can dependency manage the Avro version they want and need.
- In light of that, actually retaining Avro/Pojo backwards compatibility
seems super complicated. I would almost suggest to not do that, because it
seems it will not be transparent for users anyways (with Avro itself breaking
compatibility).
---
