Hi Danny, Thank you for bringing this up. I agree with points made by Ahmed, the split into different repositories for connectors/connector groups adds flexibility to evolve connectors without affecting other connectors.
I am also in favor of dropping the Flink version component, although this will require maintaining a compatibility matrix. As pointed out by Sergey, nightly/weekly CI builds in connector repos already indicate compatibility issues in unreleased (SNAPSHOT) versions, e.g. [1]. 1 - https://issues.apache.org/jira/browse/FLINK-34260 Thanks, Aleksandr On Tue, 11 Jun 2024 at 13:54, Ahmed Hamdy <hamdy10...@gmail.com> wrote: > Hi Danny, > Thanks for bringing this up, I might haven't driven a connector release > myself but I echo the pain and delay in releases for adding Flink version > support. > I am not really with the mono-repo approach for the following reasons > 1- We will lose the flexibility we currently have for connectors (I mean we > even had a major release 4.X for AWS connectors IIRC). > 2- We will be undoing the gains for CI decoupling, the frequency of > contributions in connectors like kafka and AWS are unmatched for others > like GCP and RabbitMQ, with new connector added and major feature work I > see unnecessary cost and delays for CI due to monorepo. > > I believe the only benefit of this approach (over the other proposed )would > be forcing adopting common connectors changes to all connectors at once > instead of relying on divided efforts to port the change, however for the > reasons mentioned above I still wouldn't vote for this approach. > > I am in favor of dropping the version though, I understand it might be > confusing for users but as Sergey mentioned; many times the changes of a > new Flink version don't introduce compatibility issues to connectors so I > believe I believe it might be an easier task than It sound initially. > > A question would be what do you think the best approach to when we do > introduce backward compatible changes to connectors API like in this PR[1], > in this case existing connectors would still work with the newly released > Flink version but would rather accumulate technical debt and removing it > would be an Adhoc task for maintainers which I believe is an accepted > tradeoff but would love to hear the feedback. > > 1- > > https://github.com/apache/flink/pull/24180/files#diff-2ffade463560e5941912b91b12a07c888313a4cc7e39ca8700398ed6975b8e90 > > Best Regards > Ahmed Hamdy > > > On Tue, 11 Jun 2024 at 08:50, Sergey Nuyanzin <snuyan...@gmail.com> wrote: > > > Thanks for starting this discussion Danny > > > > I will put my 5 cents here > > > > From one side yes, support of new Flink release takes time as it was > > mentioned above > > However from another side most of the connectors (main/master branches) > > supported Flink 1.19 > > even before it was released, same for 1.20 since they were testing > against > > master and supported version branches. > > There are already nightly/weekly jobs (depending on connector) > > running against the latest Flink SNAPSHOTs. And it has already helped to > > catch some blocker issues like[1], [2]. > > In fact there are more, I need to spend time retrieving all of them. > > > > I would also not vote for connector mono-repo release since we recently > > just splitted it. > > > > The thing I would suggest: > > since we already have nightly/weekly jobs for connectors testing against > > Flink main repo master branch > > we could add a requirement before the release of Flink itself having > these > > job results also green. > > > > [1] https://issues.apache.org/jira/browse/FLINK-34941 > > [2] https://issues.apache.org/jira/browse/FLINK-32978#comment-17804459 > > > > On Tue, Jun 11, 2024 at 8:24 AM Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > Thanks for bringing this up, Danny. This is indeed an important issue > > that > > > the community needs to improve on. > > > > > > Personally, I think a mono-repo might not be a bad idea, if we apply > > > different rules for the connector releases. To be specific: > > > - flink-connectors 1.19.x contains all connectors that are compatible > > with > > > Flink 1.19.x. > > > - allow not only bug-fixes, but also new features for a third-digit > > release > > > (e.g., flink-connectors 1.19.1) > > > > > > This would allow us to immediately release flink-connectors 1.19.0 > right > > > after flink 1.19.0 is out, excluding connectors that are no longer > > > compatible with flink 1.19. Then we can have a couple of > flink-connectors > > > 1.19.x releases, gradually adding the missing connectors back. In the > > worst > > > case, this would result in as many releases as having separated > connector > > > repose. The benefit comes from 1) there are chances to combine > releasing > > of > > > multiple connectors into one release of the mono repo (if they are > ready > > > around the same time), and 2) no need to maintain a compatibility > matrix > > > and worrying about it being out-of-sync with the code base. > > > > > > However, one thing I don't like about this approach is that it requires > > > combining all the repos we just separated from the main-repo to another > > > mono-repo. That back-and-forth is annoying. So I'm just speaking out my > > > ideas, but would not strongly insist on this. > > > > > > And big +1 for compatibility tools and ci checks. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Tue, Jun 11, 2024 at 2:38 AM David Radley <david_rad...@uk.ibm.com> > > > wrote: > > > > > > > Hi Danny, > > > > I think your proposal is a good one. This is the approach that we > took > > > > with the Egeria project, firstly taking the connectors out of the > main > > > > repo, then connectors having their own versions that incremented > > > > organically rather then tied to the core release. > > > > > > > > Blue sky thinking - I wonder if we could : > > > > - have a wizard / utility so the user inputs which Flink level they > > want > > > > and which connectors; the utility knows the compatibility matrix and > > > > downloads the appropriate bundles. > > > > - have the docs interrogate the core and connector repos to check the > > > poms > > > > for the Flink levels and the pr builds to have ?live? docs showing > the > > > > supported Flink levels. PyTorch does something like this for it?s > docs. > > > > > > > > Kind regards, David. > > > > > > > > > > > > > > > > From: Danny Cranmer <dannycran...@apache.org> > > > > Date: Monday, 10 June 2024 at 17:26 > > > > To: dev <dev@flink.apache.org> > > > > Subject: [EXTERNAL] [DISCUSS] Connector Externalization Retrospective > > > > Hello Flink community, > > > > > > > > It has been over 2 years [1] since we started externalizing the Flink > > > > connectors to dedicated repositories from the main Flink code base. > The > > > > past discussions can be found here [2]. The community decided to > > > > externalize the connectors to primarily 1/ improve stability and > speed > > of > > > > the CI, and 2/ decouple version and release lifecycle to allow the > > > projects > > > > to evolve independently. The outcome of this has resulted in each > > > connector > > > > requiring a dedicated release per Flink minor version, which is a > > burden > > > on > > > > the community. Flink 1.19.0 was released on 2024-03-18 [3], the first > > > > supported connector followed roughly 2.5 months later on 2024-06-06 > [4] > > > > (MongoDB). There are still 5 connectors that do not support Flink > 1.19 > > > [5]. > > > > > > > > Two decisions contribute to the high lag between releases. 1/ > creating > > > one > > > > repository per connector instead of a single flink-connector > mono-repo > > > and > > > > 2/ coupling the Flink version to the connector version [6]. A single > > > > connector repository would reduce the number of connector releases > > from N > > > > to 1, but would couple the connector CI and reduce release > flexibility. > > > > Decoupling the connector versions from Flink would eliminate the need > > to > > > > release each connector for each new Flink minor version, but we would > > > need > > > > a new compatibility mechanism. > > > > > > > > I propose that from each next connector release we drop the coupling > on > > > the > > > > Flink version. For example, instead of 3.4.0-1.20 > (<connector>.<flink>) > > > we > > > > would release 3.4.0 (<connector>). We can model a compatibility > matrix > > > > within the Flink docs to help users pick the correct versions. This > > would > > > > mean we would usually not need to release a new connector version per > > > Flink > > > > version, assuming there are no breaking changes. Worst case, if > > breaking > > > > changes impact all connectors we would still need to release all > > > > connectors. However, for Flink 1.17 and 1.18 there were only a > handful > > of > > > > issues (breaking changes), and mostly impacting tests. We could > decide > > to > > > > align this with Flink 2.0, however I see no compelling reason to do > so. > > > > This was discussed previously [2] as a long term goal once the > > connector > > > > APIs are stable. But I think the current compatibility rules support > > this > > > > change now. > > > > > > > > I would prefer to not create a connector mono-repo. Separate repos > > gives > > > > each connector more flexibility to evolve independently, and removing > > > > unnecessary releases will significantly reduce the release effort. > > > > > > > > I would like to hear opinions and ideas from the community. In > > > particular, > > > > are there any other issues you have observed that we should consider > > > > addressing? > > > > > > > > Thanks, > > > > Danny. > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/flink-connector-elasticsearch/commit/3ca2e625e3149e8864a4ad478773ab4a82720241 > > > > [2] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h > > > > [3] > > > > > > > > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > [4] https://flink.apache.org/downloads/#apache-flink-connectors-1 > > > > [5] https://issues.apache.org/jira/browse/FLINK-35131 > > > > [6] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Examples > > > > > > > > Unless otherwise stated above: > > > > > > > > IBM United Kingdom Limited > > > > Registered in England and Wales with number 741598 > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 > 3AU > > > > > > > > > > > > > -- > > Best regards, > > Sergey > > >
