Thanks for starting this discussion Danny I will put my 5 cents here
>From one side yes, support of new Flink release takes time as it was mentioned above However from another side most of the connectors (main/master branches) supported Flink 1.19 even before it was released, same for 1.20 since they were testing against master and supported version branches. There are already nightly/weekly jobs (depending on connector) running against the latest Flink SNAPSHOTs. And it has already helped to catch some blocker issues like[1], [2]. In fact there are more, I need to spend time retrieving all of them. I would also not vote for connector mono-repo release since we recently just splitted it. The thing I would suggest: since we already have nightly/weekly jobs for connectors testing against Flink main repo master branch we could add a requirement before the release of Flink itself having these job results also green. [1] https://issues.apache.org/jira/browse/FLINK-34941 [2] https://issues.apache.org/jira/browse/FLINK-32978#comment-17804459 On Tue, Jun 11, 2024 at 8:24 AM Xintong Song <tonysong...@gmail.com> wrote: > Thanks for bringing this up, Danny. This is indeed an important issue that > the community needs to improve on. > > Personally, I think a mono-repo might not be a bad idea, if we apply > different rules for the connector releases. To be specific: > - flink-connectors 1.19.x contains all connectors that are compatible with > Flink 1.19.x. > - allow not only bug-fixes, but also new features for a third-digit release > (e.g., flink-connectors 1.19.1) > > This would allow us to immediately release flink-connectors 1.19.0 right > after flink 1.19.0 is out, excluding connectors that are no longer > compatible with flink 1.19. Then we can have a couple of flink-connectors > 1.19.x releases, gradually adding the missing connectors back. In the worst > case, this would result in as many releases as having separated connector > repose. The benefit comes from 1) there are chances to combine releasing of > multiple connectors into one release of the mono repo (if they are ready > around the same time), and 2) no need to maintain a compatibility matrix > and worrying about it being out-of-sync with the code base. > > However, one thing I don't like about this approach is that it requires > combining all the repos we just separated from the main-repo to another > mono-repo. That back-and-forth is annoying. So I'm just speaking out my > ideas, but would not strongly insist on this. > > And big +1 for compatibility tools and ci checks. > > Best, > > Xintong > > > > On Tue, Jun 11, 2024 at 2:38 AM David Radley <david_rad...@uk.ibm.com> > wrote: > > > Hi Danny, > > I think your proposal is a good one. This is the approach that we took > > with the Egeria project, firstly taking the connectors out of the main > > repo, then connectors having their own versions that incremented > > organically rather then tied to the core release. > > > > Blue sky thinking - I wonder if we could : > > - have a wizard / utility so the user inputs which Flink level they want > > and which connectors; the utility knows the compatibility matrix and > > downloads the appropriate bundles. > > - have the docs interrogate the core and connector repos to check the > poms > > for the Flink levels and the pr builds to have ?live? docs showing the > > supported Flink levels. PyTorch does something like this for it?s docs. > > > > Kind regards, David. > > > > > > > > From: Danny Cranmer <dannycran...@apache.org> > > Date: Monday, 10 June 2024 at 17:26 > > To: dev <dev@flink.apache.org> > > Subject: [EXTERNAL] [DISCUSS] Connector Externalization Retrospective > > Hello Flink community, > > > > It has been over 2 years [1] since we started externalizing the Flink > > connectors to dedicated repositories from the main Flink code base. The > > past discussions can be found here [2]. The community decided to > > externalize the connectors to primarily 1/ improve stability and speed of > > the CI, and 2/ decouple version and release lifecycle to allow the > projects > > to evolve independently. The outcome of this has resulted in each > connector > > requiring a dedicated release per Flink minor version, which is a burden > on > > the community. Flink 1.19.0 was released on 2024-03-18 [3], the first > > supported connector followed roughly 2.5 months later on 2024-06-06 [4] > > (MongoDB). There are still 5 connectors that do not support Flink 1.19 > [5]. > > > > Two decisions contribute to the high lag between releases. 1/ creating > one > > repository per connector instead of a single flink-connector mono-repo > and > > 2/ coupling the Flink version to the connector version [6]. A single > > connector repository would reduce the number of connector releases from N > > to 1, but would couple the connector CI and reduce release flexibility. > > Decoupling the connector versions from Flink would eliminate the need to > > release each connector for each new Flink minor version, but we would > need > > a new compatibility mechanism. > > > > I propose that from each next connector release we drop the coupling on > the > > Flink version. For example, instead of 3.4.0-1.20 (<connector>.<flink>) > we > > would release 3.4.0 (<connector>). We can model a compatibility matrix > > within the Flink docs to help users pick the correct versions. This would > > mean we would usually not need to release a new connector version per > Flink > > version, assuming there are no breaking changes. Worst case, if breaking > > changes impact all connectors we would still need to release all > > connectors. However, for Flink 1.17 and 1.18 there were only a handful of > > issues (breaking changes), and mostly impacting tests. We could decide to > > align this with Flink 2.0, however I see no compelling reason to do so. > > This was discussed previously [2] as a long term goal once the connector > > APIs are stable. But I think the current compatibility rules support this > > change now. > > > > I would prefer to not create a connector mono-repo. Separate repos gives > > each connector more flexibility to evolve independently, and removing > > unnecessary releases will significantly reduce the release effort. > > > > I would like to hear opinions and ideas from the community. In > particular, > > are there any other issues you have observed that we should consider > > addressing? > > > > Thanks, > > Danny. > > > > [1] > > > > > https://github.com/apache/flink-connector-elasticsearch/commit/3ca2e625e3149e8864a4ad478773ab4a82720241 > > [2] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h > > [3] > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > [4] https://flink.apache.org/downloads/#apache-flink-connectors-1 > > [5] https://issues.apache.org/jira/browse/FLINK-35131 > > [6] > > > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Examples > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > > -- Best regards, Sergey
