Hi,
this has come up a few times now and I think we need to discuss the
guarantees that we want to officially give for this. What I mean by
cross-version compatibility is using, say, a Flink 1.10 Kafka connector
dependency/jar with Flink 1.11, or a Flink 1.10.0 connector with Flink
1.10.1. In the past, this hast mostly worked. I think this was largely
by accident, though.
The problem is that connectors, which might be annotated as @Public can
use classes internally which are annotated as @Internal or
@PublicEvolving. If those internal dependencies change, then the
connector jar can not be used for different versions. This has happened
at least in [1] and [2], where [2] was caused by the interplay of [3]
and [4]. The initial release note on [4] was saying that the Kafka 0.9
connector can be used with Flink 1.11, but this was rendered wrong by
[3]. (Also, sorry for all the []s ...)
What should we do about it? So far our strategy for ensuring that
jars/dependencies are compatible between versions was "hope". If we want
to really verify this compatibility we would have to ensure that
"public" code does not transitively use any "non-public" dependencies.
An alternative would be to say we don't support any cross-version
compatibility between Flink versions. If users want to use an older
connector they would have to copy the code, make sure it compiles
against a newer Flink version and then manage that themselves
What do you think?
Best,
Aljoscha
[1] https://issues.apache.org/jira/browse/FLINK-13586
[2] https://github.com/apache/flink/pull/12699#discussion_r442013514
[3] https://issues.apache.org/jira/browse/FLINK-17376
[4] https://issues.apache.org/jira/browse/FLINK-15115
