Re: [DISCUSS] Connector Externalization Retrospective

2024-06-13 Thread Danny Cranmer
Thanks all for the feedback. @David > have a wizard / utility so the user inputs which Flink level they want and which connectors; the utility knows the compatibility matrix and downloads the appropriate bundles. My colleagues developed a Maven plugin [1] that performs static checks. Something l

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Thomas Weise
Thanks for bringing this discussion back. When we decided to decouple the connectors, we already discussed that we will only realize the full benefit when the connectors actually become independent from the Flink minor releases. Until that happens we have a ton of extra work but limited gain. Base

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Chesnay Schepler
On 10/06/2024 18:25, Danny Cranmer wrote: This would mean we would usually not need to release a new connector version per Flink version, assuming there are no breaking changes. We technically can't do this because we don't provide binary compatibility across minor versions. That's the entire re

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Aleksandr Pilipenko
> > > > with the Egeria project, firstly taking the connectors out of the > main > > > > repo, then connectors having their own versions that incremented > > > > organically rather then tied to the core release. > > > > > > > > Blue sky thinking -

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Muhammet Orazov
Hello Danny, Thanks for the starting the discussion. -1 for mono-repo, and -+1 for dropping Flink version. I have mixed opinion with dropping the Flink version. Usually, large production migrations happen on Flink versions and users want also naturally update the connectors compatible for that

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Ahmed Hamdy
zard / utility so the user inputs which Flink level they > want > > > and which connectors; the utility knows the compatibility matrix and > > > downloads the appropriate bundles. > > > - have the docs interrogate the core and connector repos to check the > > pom

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-11 Thread Sergey Nuyanzin
atrix and > > downloads the appropriate bundles. > > - have the docs interrogate the core and connector repos to check the > poms > > for the Flink levels and the pr builds to have ?live? docs showing the > > supported Flink levels. PyTorch does something like this for i

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-10 Thread Xintong Song
> for the Flink levels and the pr builds to have ?live? docs showing the > supported Flink levels. PyTorch does something like this for it?s docs. > > Kind regards, David. > > > > From: Danny Cranmer > Date: Monday, 10 June 2024 at 17:26 > To: dev > Subject

Re: [DISCUSS] Connector Externalization Retrospective

2024-06-10 Thread David Radley
pr builds to have ?live? docs showing the supported Flink levels. PyTorch does something like this for it?s docs. Kind regards, David. From: Danny Cranmer Date: Monday, 10 June 2024 at 17:26 To: dev Subject: [EXTERNAL] [DISCUSS] Connector Externalization Retrospective Hello Flink community

[DISCUSS] Connector Externalization Retrospective

2024-06-10 Thread Danny Cranmer
Hello Flink community, It has been over 2 years [1] since we started externalizing the Flink connectors to dedicated repositories from the main Flink code base. The past discussions can be found here [2]. The community decided to externalize the connectors to primarily 1/ improve stability and spe