Thanks all for the feedback. @David
> have a wizard / utility so the user inputs which Flink level they want and which connectors; the utility knows the compatibility matrix and downloads the appropriate bundles. My colleagues developed a Maven plugin [1] that performs static checks. Something like this might work, however it requires users to actually use it, and keep it up to date. We could provide a Flink Maven bom or similar that manages the dependencies on their behalf. @Xington > This would allow us to immediately release flink-connectors 1.19.0 right after flink 1.19.0 is out, excluding connectors that are no longer compatible with flink 1.19. While this is convenient, this coupling is something we were trying to avoid. With this approach we cannot make breaking changes without waiting for a Flink major release. @Sergey > The thing I would suggest: since we already have nightly/weekly jobs for connectors testing against Flink main repo master branch we could add a requirement before the release of Flink itself having these job results also green. It depends how they get green. If we change the connector code to get it green, this implies the last connector version does not support the new Flink version and we require a connector release. This is good information to have, but still results in a connector release. @Muhammet > I have mixed opinion with dropping the Flink version. Usually, large production migrations happen on Flink versions and users want also naturally update the connectors compatible for that Flink version. I agree it is less convenient to have arbitrary versions, however the version coupling does not scale well for the community. We already follow the version decoupling strategy for other Flink projects, such as Statefun, Flink ML and the Flink Kubernetes Operator [2]. @Ahmed > A question would be what do you think the best approach to when we do introduce backward compatible changes to connectors API like in this PR, in this case existing connectors would still work with the newly released Flink version but would rather accumulate technical debt and removing it would be an Adhoc task for maintainers which I believe is an accepted tradeoff but would love to hear the feedback. I would not change our current process. Create a Jira task to update each connector, do the work, and then this is included as part of the next connector release. I am not sure how this is impacted by decoupling versions or monorepo discussions. @Chesnay > We technically can't do this because we don't provide binary compatibility across minor versions. We can still perform the same compatibility checks we do today until we achieve full backwards compatibility. Currently we perform these checks and then do a heavyweight version release. The new process would be gated by the compatibility matrix update. Therefore we can still perform the same compatibility checks, however when the checks pass, the process of updating the compatibility matrix is much lighter than a release. Once we achieve full binary compatibility this will increase our confidence and allow an even more streamlined process. For example, the compatibility matrix might say "supports 1.19+", rather than "supports 1.19". > That's the entire reason we did this coupling in the first place, and imo /we/ shouldn't take a shortcut but still have our users face that very problem. I would think of this as an incremental improvement rather than a shortcut. I agree the user experience is not as nice having to look at a compatibility matrix vs encoded in the version. However, more timely connector support outweighs this in my opinion. By gating the compatibility matrix update by our compatibility checks we can provide the same level of compatibility guarantees we do today. > We knew this was gonna by annoying for us; that was intentional and meant to finally push us towards binary compatibility /guarantees/. The lag between core and Flink releases is the biggest problem at the moment. However, it is annoying, yes it is. @Thomas > Would it be possible to verify that by running e2e tests of connector binaries built against an earlier Flink minor version against the latest Flink minor release candidate as part of the release? We already build all connectors against the next Flink snapshot in the nightly/weekly builds. So we can/do get early sight of breaking changes/incompatibilities. Thanks, Danny [1] https://github.com/awslabs/static-checker-flink [2] https://flink.apache.org/downloads/ On Tue, Jun 11, 2024 at 8:12 PM Thomas Weise <t...@apache.org> wrote: > Thanks for bringing this discussion back. > > When we decided to decouple the connectors, we already discussed that we > will only realize the full benefit when the connectors actually become > independent from the Flink minor releases. Until that happens we have a ton > of extra work but limited gain. Based on the assumption that getting to the > binary compatibility guarantee is our goal - not just for the connectors > managed within the Flink project but for the ecosystem as a whole - I don't > see the benefit of mono repo or similar approach that targets the symptom > rather than the cause. > > In the final picture we would only need connector releases if/when a > specific connector changes and the repository per connector layout would > work well. > > I also agree with Danny that we may not have to wait for Flink 2.0 for > that. How close are we to assume compatibility of the API surface that > affects connectors? It appears that practically there have been little to > no known issues in the last couple of releases? Would it be possible to > verify that by running e2e tests of connector binaries built against an > earlier Flink minor version against the latest Flink minor release > candidate as part of the release? > > Thanks, > Thomas > > > On Tue, Jun 11, 2024 at 11:05 AM Chesnay Schepler <ches...@apache.org> > wrote: > > > On 10/06/2024 18:25, Danny Cranmer wrote: > > > This would > > > mean we would usually not need to release a new connector version per > > Flink > > > version, assuming there are no breaking changes. > > We technically can't do this because we don't provide binary > > compatibility across minor versions. > > That's the entire reason we did this coupling in the first place, and > > imo /we/ shouldn't take a shortcut but still have our users face that > > very problem. > > We knew this was gonna by annoying for us; that was intentional and > > meant to finally push us towards binary compatibility /guarantees/. >
