Thanks for bringing this up, Danny. This is indeed an important issue that the community needs to improve on.
Personally, I think a mono-repo might not be a bad idea, if we apply different rules for the connector releases. To be specific: - flink-connectors 1.19.x contains all connectors that are compatible with Flink 1.19.x. - allow not only bug-fixes, but also new features for a third-digit release (e.g., flink-connectors 1.19.1) This would allow us to immediately release flink-connectors 1.19.0 right after flink 1.19.0 is out, excluding connectors that are no longer compatible with flink 1.19. Then we can have a couple of flink-connectors 1.19.x releases, gradually adding the missing connectors back. In the worst case, this would result in as many releases as having separated connector repose. The benefit comes from 1) there are chances to combine releasing of multiple connectors into one release of the mono repo (if they are ready around the same time), and 2) no need to maintain a compatibility matrix and worrying about it being out-of-sync with the code base. However, one thing I don't like about this approach is that it requires combining all the repos we just separated from the main-repo to another mono-repo. That back-and-forth is annoying. So I'm just speaking out my ideas, but would not strongly insist on this. And big +1 for compatibility tools and ci checks. Best, Xintong On Tue, Jun 11, 2024 at 2:38 AM David Radley <david_rad...@uk.ibm.com> wrote: > Hi Danny, > I think your proposal is a good one. This is the approach that we took > with the Egeria project, firstly taking the connectors out of the main > repo, then connectors having their own versions that incremented > organically rather then tied to the core release. > > Blue sky thinking - I wonder if we could : > - have a wizard / utility so the user inputs which Flink level they want > and which connectors; the utility knows the compatibility matrix and > downloads the appropriate bundles. > - have the docs interrogate the core and connector repos to check the poms > for the Flink levels and the pr builds to have ?live? docs showing the > supported Flink levels. PyTorch does something like this for it?s docs. > > Kind regards, David. > > > > From: Danny Cranmer <dannycran...@apache.org> > Date: Monday, 10 June 2024 at 17:26 > To: dev <dev@flink.apache.org> > Subject: [EXTERNAL] [DISCUSS] Connector Externalization Retrospective > Hello Flink community, > > It has been over 2 years [1] since we started externalizing the Flink > connectors to dedicated repositories from the main Flink code base. The > past discussions can be found here [2]. The community decided to > externalize the connectors to primarily 1/ improve stability and speed of > the CI, and 2/ decouple version and release lifecycle to allow the projects > to evolve independently. The outcome of this has resulted in each connector > requiring a dedicated release per Flink minor version, which is a burden on > the community. Flink 1.19.0 was released on 2024-03-18 [3], the first > supported connector followed roughly 2.5 months later on 2024-06-06 [4] > (MongoDB). There are still 5 connectors that do not support Flink 1.19 [5]. > > Two decisions contribute to the high lag between releases. 1/ creating one > repository per connector instead of a single flink-connector mono-repo and > 2/ coupling the Flink version to the connector version [6]. A single > connector repository would reduce the number of connector releases from N > to 1, but would couple the connector CI and reduce release flexibility. > Decoupling the connector versions from Flink would eliminate the need to > release each connector for each new Flink minor version, but we would need > a new compatibility mechanism. > > I propose that from each next connector release we drop the coupling on the > Flink version. For example, instead of 3.4.0-1.20 (<connector>.<flink>) we > would release 3.4.0 (<connector>). We can model a compatibility matrix > within the Flink docs to help users pick the correct versions. This would > mean we would usually not need to release a new connector version per Flink > version, assuming there are no breaking changes. Worst case, if breaking > changes impact all connectors we would still need to release all > connectors. However, for Flink 1.17 and 1.18 there were only a handful of > issues (breaking changes), and mostly impacting tests. We could decide to > align this with Flink 2.0, however I see no compelling reason to do so. > This was discussed previously [2] as a long term goal once the connector > APIs are stable. But I think the current compatibility rules support this > change now. > > I would prefer to not create a connector mono-repo. Separate repos gives > each connector more flexibility to evolve independently, and removing > unnecessary releases will significantly reduce the release effort. > > I would like to hear opinions and ideas from the community. In particular, > are there any other issues you have observed that we should consider > addressing? > > Thanks, > Danny. > > [1] > > https://github.com/apache/flink-connector-elasticsearch/commit/3ca2e625e3149e8864a4ad478773ab4a82720241 > [2] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h > [3] > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > [4] https://flink.apache.org/downloads/#apache-flink-connectors-1 > [5] https://issues.apache.org/jira/browse/FLINK-35131 > [6] > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Examples > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
