Hi Arvid, Thanks for the information. I think your plan makes sense given that we want to get a Flink 2.0 compatible connector out ASAP.
I can keep updating my Kafka 4.0.0 PR [1] until we are ready to do the release containing it. In my opinion version numbers are not a constrained resource and given that moving to the 4.x Kafka clients will drop support for Kafka 2.0 and older, a 5.0 release of the connector might make sense. It would also have the advantage of moving the connector's version number out of the range of the Kafka versions, which would reduce confusion on whether the connector version equates to the supported Kafka version. However, that conversation can wait until later. I am happy to push ahead with a 4.0 connector release with Flink 2.0.0 and Kafka 3.9.0. Tom Cooper @tomcooper.dev | https://tomcooper.dev [1] https://github.com/apache/flink-connector-kafka/pull/161 On Tuesday, 1 April 2025 at 21:23, Arvid Heise <ar...@apache.org> wrote: > Hi Tom, > > thanks for driving this. > > Updating the Kafka client library has been done only occasionally and > there was no formal process. It makes sense to give stronger > guidelines. I guess a rigid process would only make sense if we > enforce it for example through bots and that was highly debated a > couple of years ago in Apache projects (I would be very much in favor > of getting bot support for dependencies). > > Here is my proposal: > * Kafka-connector-4.0.0 uses kafka-client-3.9.0 because it's the > latest minor release in the 3.X. I went over the things that they > (plan to) fix in 3.9.1 and nothing major pops out. 3.9.0 gives folks > the option to relatively effortless downgrade to 3.8.1 in DataStreams > (it's just a transitive dependency). For SQL, folks are bound to 3.9.0 > unless they rebuild the module from scratch, which is also a moderate > effort. No code change is required. > * Kafka-connector-3.5.0 with a kafka-client-3.9.0 for Flink 1.X also > makes a lot of sense if Kafka-connector-4.0.0 cannot run on Flink 1.X > (to confirm). Else, folks can just use Kafka-connector-4.X. > * Release Kafka-connector-4.0.1 once kafka-client-3.9.1 is out + any > bug fix that is needed in our code. > * Release Kafka-connector-4.1.0 once kafka-client-4.0.1 or 4.1.0 is > out. In my experience, the very first version of a major release is > usually for early adopters only. Most of the new features in that > release are only available once you also update brokers to 4.0.0, > which will probably take a longer time for most setups. Of course, the > effort can be expedited if there is actual demand for > kafka-client-4.0.0. > > As Yanquan mentioned, the biggest need is currently on > Kafka-connector-4.0.0. We need to find a release manager asap and get > that release out next week (so probably start this week with cut and > voting). We need to bundle our resources on that. > > Once that is done, we should look at Kafka-connector-3.5.0 release. > Then, the others. > > Note that I don't see the need for another major release to bump > kafka-client to 4.0.0. From the user's perspective, this change is > mostly transparent. I think that release basically just cuts off some > very old broker versions <3.0. I'm assuming most folks are on 3.5+ > atm. In contrast, for kafka-connector-4.0.0, Yanquan deleted 22k LOC > to drop all the legacy consumer/producer. > > LMK what you think about that proposal. > > Arvid > > On Tue, Apr 1, 2025 at 6:10 PM Yanquan Lv decq12y...@gmail.com wrote: > > > Hi, Cooper. Thank you for bringing up the discussion about Kafka 4.0.0. > > I tend to release Kafka Connector 4.0 with Flink 2.0.0 and Kafka 3.9.0, or > > Kafka 3.8.1 if If the community is concerned about 3.9.0. > > > > Here are some of my opinions: > > > > 1. The plan for Kafka Connector 4.0.0 was proposed[1] a long time ago, > > mainly to support the Flink 2.0 API. Although it has been delayed for some > > time, users' demand for this version is becoming increasingly strong (as we > > can see from emails or Slack), so it is more important to release a version > > that supports Flink 2.0 as soon as possible. > > > > 2. For Bump to Kafka 4.0.0, I personally believe that this is a decision > > that requires careful analysis, because rolling back to Kafka client > > version 3. x will require additional workload, and the impact on > > performance and API compatibility may not be reflected solely through IT > > cases(Of course, I believe it will have a positive impact). > > I don't recommend making two major changes in one version,So perhaps > > 4.1/5.0 is more suitable for doing this. > > > > 3. As for Kafka 3.9.0, I don't have a big opinion as we have been using > > Kafka client 3.4.0 for almost two years since FLINK-31599[2]. Of course, I > > also approve of using Kafka 3.8.1. > > But for using 3.9.1, the benefits are not very significant and it will > > block the activities of other developers who rely on Flink connector Kafka > > release 4.0. As I know, some e2e in Flink rely on Kafka connectors, but > > they have been temporarily removed[3], and development activities[4] for > > Apache Paimon adaptation to Flink 2.0, etc. > > > > Best, Yanquan. > > > > [1]https://lists.apache.org/thread/rl7prqop7wfn2o8j2j9fd96dgr1bjjnx > > [2]https://issues.apache.org/jira/browse/FLINK-31599 > > [3]https://issues.apache.org/jira/browse/FLINK-36413 > > [4]https://github.com/apache/paimon/issues/5350 > > > > > 2025年4月1日 22:50，Tom Cooper c...@tomcooper.dev 写道： > > > > > > Kafka 4.0.0
