Hi Chesnay, My understanding is that source compatibility is the initial step towards a stronger guarantee that will reduce the pain downstream. In that spirit, I would anticipate that we are not taking steps to make the long term goal harder to achieve?
The FLIP [1] states: "There is no official guarantee that a program compiled against an earlier version can be executed on a newer Flink cluster (no ABI backwards compatibility). But eventually we should try provide this guarantee." Cheers, Thomas [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%3A+Source+API+stability+guarantees On Fri, Feb 11, 2022 at 12:26 AM Chesnay Schepler <ches...@apache.org> wrote: > > The conclusion in FLIP-196 was that we only provide source compatibility for > Public(Evolving) APIs, which means that _everything_ must be compiled against > the Flink version you are using it with. > > Doesn't that mean that such dependency conflicts are then user errors? > > On 10/02/2022 20:19, Thomas Weise wrote: > > Hi Alexander, > > It is beneficial for users to be able to replace/choose a connector as > part of their application. When flink-connector-base is included in > dist, that becomes more difficult. We can run into hard to understand > dependency conflicts such as [1]. Depending on the Flink platform > used, it may also be hard to update something that is part of dist. I > would prefer to keep the module outside dist. > > Thanks, > Thomas > > [1] https://lists.apache.org/thread/0ht5y2tyzpt16ft36zm428182dxfs3zx > > On Wed, Feb 9, 2022 at 3:26 AM Alexander Fedulov > <alexan...@ververica.com> wrote: > > Hi everyone, > > I would like to discuss the best approach to address the issue raised > in FLINK-25927 [1]. It can be summarized as follows: > > flink-connector-base is currently inconsistently used in connectors > > (directly shaded in some and transitively pulled in via > flink-connector-files which is itself shaded in the table uber jar) > > FLINK-24687 [2] moved flink-connector-files out of the flink-table uber > > jar > > It is necessary to make usage of flink-connector-base consistent across > > all connectors > > One approach is to stop shading flink-connector-files in all connectors and > instead package it in flink-dist, making it a part of Flink-wide provided > public API. This approach is investigated in the following PoC PR: 18545 > [3]. The issue with this approach is that it breaks any existing CI and > IDE setups that do not directly rely on flink-dist and also do not include > flink-connector-files as an explicit dependency. > > In theory, a nice alternative would be to make it a part of a dependency > that is ubiquitously provided, for instance, flink-streaming-java. Doing > that for flink-streaming-java would, however, introduce a dependency cycle > and is currently not feasible. > > It would be great to hear your opinions on what could be the best way > forward here. > > [1] https://issues.apache.org/jira/browse/FLINK-25927 > [2] https://issues.apache.org/jira/browse/FLINK-24687 > [3] https://github.com/apache/flink/pull/18545 > > > Thanks, > Alexander Fedulov > >
