Hi Tom,
Thinking about releases. I see:
Kafka connector 3.3 and 3.4, Flink 1.20
Kafka connector 4.0, Kafka client 3.9.0 Flink 2.0
Kafka connector 4.0.1, Kafka client 3.9.1 Flink 2.0. (does this version work
with Flink 2.1 or is [1] a show stopper?)
I assume we might want:
Kafka connector 4.0.2, Kafka client 3.9.1 Flink 2.1
Kafka connector 5 Kafka client 4.0 Flink 2.1
This would be a lot of concurrent releases that are supported,
Kind regards, David.
[1] https://github.com/apache/flink-connector-kafka/pull/187
From: Tom Cooper <c...@tomcooper.dev>
Date: Friday, 29 August 2025 at 09:57
To: dev@flink.apache.org <dev@flink.apache.org>
Subject: [EXTERNAL] Re: Flink Connector Kafka Releases
FYI, I have posted a PR [1] for the Flink 2.1 update for the Kafka connector.
The main change with this update is that Flink has removed [2] (without
deprecating first) the stripRowPrefix method from the DataTypeUtilsTest class.
I couldn't see any evidence that this logic was moved to any other public API
so this PR adds that logic into a util class within the connector.
Also note that with the update to Flink 2.1.0 the Kafka connector drops support
for Python 3.8, I added testing for Python 3.11 but 3.12-cython (Flink 2.1
added support for Python 3.12) is not yet available.
If we can get his PR merged and the Kafka 4.0.0 update PR [3], we will be in a
good position for the 5.0.0 release?
Thanks,
Tom Cooper
@tomcooper.dev | https://tomcooper.dev
[1] https://github.com/apache/flink-connector-kafka/pull/187
[2] https://github.com/apache/flink/pull/26784
On Thursday, 14 August 2025 at 21:59, Weiqing Yang <yangweiqing...@gmail.com>
wrote:
> Thanks, Fabian. This is helpful to know.
>
> On Thu, Aug 14, 2025 at 1:50 AM Fabian Paul fp...@apache.org wrote:
>
> > Hi Weiqing,
> >
> > So far, there is no concrete timeline to publish the Kafka 5.0
> > release. We are still releasing 4.0.1 at the moment. Regarding the
> > Flink 2.1 support, you should be able to use the flink kafka connector
> > that supports 2.0 also with 2.1. The underlying connector library is
> > stable across minor versions. This also means that a new Flink minor
> > release doesn't necessarily warrant a new connector release.
> >
> > Best,
> > Fabian
> >
> > On Wed, Aug 13, 2025 at 6:57 PM Weiqing Yang yangweiqing...@gmail.com
> > wrote:
> >
> > > Hi Tom, Fabian,
> > >
> > > Thanks for the updates on the v4.0.1 release. I noticed that v4.0.1 is
> > > going out with Flink 2.0 (link
> > > <
> > > https://github.com/apache/flink-connector-kafka/blob/v4.0.1-rc2/pom.xml#L56
> > > ).
> > > Do you have a timeline or target window in mind for the Flink Connector
> > > Kafka 5.0 release that will include Flink 2.1 support?
> > >
> > > Thanks,
> > > Weiqing
> > >
> > > On Mon, Aug 11, 2025 at 7:41 AM Tom Cooper c...@tomcooper.dev wrote:
> > >
> > > > Hi Fabian,
> > > >
> > > > Sorry, for the late reply, this message somehow ended up in my spam
> > > > filter!?
> > > >
> > > > I think having the Flink 2.1 upgrade included in the move to Flink
> > > > Connector Kafka 5.0 makes sense.
> > > > I am hoping to find the time to work on the upgrade to Flink 2.1 at
> > > > end of
> > > > this week or next.
> > > > Unless, of course, you are plan to work on that?
> > > >
> > > > Regards,
> > > >
> > > > Tom Cooper
> > > > @tomcooper.dev | https://tomcooper.dev
> > > >
> > > > On Tuesday, 29 July 2025 at 09:08, Fabian Paul
> > > > fp...@confluent.io.INVALID
> > > > wrote:
> > > >
> > > > > Hi Tom,
> > > > >
> > > > > Sounds good to me, I can start with the 4.0.1 release.
> > > > > Regarding the 5.0 release, I am not super sure yet what to include.
> > > > > Since releasing always takes some effort, I would also be okay with
> > > > > doing the 5.0 release with incorporating Flink 2.1. The connector
> > > > > already offers a release that is compatible with Flink 2.0, and in
> > > > > theory, 2.1 should not introduce breaking changes that affect the
> > > > > connector.
> > > > >
> > > > > Best,
> > > > > Fabian
> > > > >
> > > > > On Mon, Jul 28, 2025 at 11:03 AM Tom Cooper c...@tomcooper.dev
> > > > > wrote:
> > > > >
> > > > > > Hi Fabian,
> > > > > >
> > > > > > You make a good point, as there are only dependency updates, a
> > > > > > 4.0.1
> > > > > > release makes more sense.
> > > > > >
> > > > > > At this point the 5.0 connector release could include the soon to
> > > > > > be
> > > > > > released Kafka 4.0.1 client libraries (the RC for that is out
> > > > > > already).
> > > > > > I assume we would want to leave the flink 2.1 upgrade to a future
> > > > > > 5.1
> > > > > > release?
> > > > > >
> > > > > > Thanks for looking at this.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Tom Cooper
> > > > > > @tomcooper.dev | https://tomcooper.dev
> > > > > >
> > > > > > On Monday, 28 July 2025 at 09:51, Fabian Paul fp...@apache.org
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > Thanks for starting this discussion. I think it's a good idea to
> > > > > > > do
> > > > > > > another 4.1.0 release before proceeding with 5.0 to offer a
> > > > > > > release
> > > > > > > with the vulnerability fixed without requiring users to upgrade
> > > > > > > to
> > > > > > > Kafka 4.0. Is there a reason you prefer to do the 4.1.0 release
> > > > > > > instead of the 4.0.1 release? I reviewed the changes between the
> > > > > > > current main and the release 4.0.0 [1], and they are mostly
> > > > > > > dependency
> > > > > > > upgrades and some fixes, but without any new features. What do
> > > > > > > you
> > > > > > > think about doing a 4.0.1 release and then kicking off 5.0.0
> > > > > > > with the
> > > > > > > Kafka client upgrade?
> > > > > > >
> > > > > > > Best,
> > > > > > > Fabian
> > > > > > >
> > > > > > > [1]
> > > > > > > https://github.com/apache/flink-connector-kafka/compare/v4.0...main
> > > > > > >
> > > > > > > On Fri, Jul 25, 2025 at 11:58 AM Tom Cooper c...@tomcooper.dev
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Bumping this thread as we are now ready to merge the Kafka
> > > > > > > > 4.0.0
> > > > > > > > client update PR [1]. This will bump the major version of the
> > > > > > > > connector to
> > > > > > > > 5.0, as we are dropping support for Kafka brokers running
> > > > > > > > version
> > > > > > > > 2.0.0 or
> > > > > > > > earlier.
> > > > > > > >
> > > > > > > > However, I still think it would be worth doing a 4.1.0 release
> > > > > > > > of
> > > > > > > > the connector (with the Kafka 3.9.1 client), before the Kafka
> > > > > > > > 4.0.0
> > > > > > > > client
> > > > > > > > update is merged.
> > > > > > > >
> > > > > > > > The current Flink Kafka Connector (4.0) has a critical CVE [2],
> > > > > > > > which is patched in the 3.9.1 Kafka client library (which the
> > > > > > > > current
> > > > > > > > main
> > > > > > > > branch of the Flink connector is using). Doing a 4.1 release of
> > > > > > > > the
> > > > > > > > connector would cover any users of older Kafka versions that
> > > > > > > > want this
> > > > > > > > CVE
> > > > > > > > patched and also give a stable release of the connector using a
> > > > > > > > point
> > > > > > > > release of the Kafka client (with all the bug fixes that
> > > > > > > > entails). This
> > > > > > > > would be a good option for users who don't want to jump
> > > > > > > > straight onto
> > > > > > > > the
> > > > > > > > new major Kafka client version.
> > > > > > > >
> > > > > > > > What do people think?
> > > > > > > >
> > > > > > > > Tom Cooper
> > > > > > > > @tomcooper.dev | https://tomcooper.dev
> > > > > > > >
> > > > > > > > [1] https://github.com/apache/flink-connector-kafka/pull/161
> > > > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2025-27817
> > > > > > > >
> > > > > > > > On Wednesday, 9 July 2025 at 09:35, Tom Cooper
> > > > > > > > c...@tomcooper.dev
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I would like to start a conversation about releases for the
> > > > > > > > > Flink Connector Kafka project.
> > > > > > > > >
> > > > > > > > > We have recently updated [0] to version 3.9.1 of the Kafka
> > > > > > > > > client library, which fixes a critical CVE [1]. With that in
> > > > > > > > > mind, I
> > > > > > > > > think
> > > > > > > > > it would be prudent to have a 4.1.0 release as soon as
> > > > > > > > > possible that
> > > > > > > > > includes this. It would also be good to include the
> > > > > > > > > dependency bumps
> > > > > > > > > from
> > > > > > > > > this PR [2] in that release.
> > > > > > > > >
> > > > > > > > > With the 4.1.0 release out, we could then move to looking at
> > > > > > > > > the
> > > > > > > > > Kafka 4.0 upgrade (there is already a PR [3] for that). The
> > > > > > > > > main point
> > > > > > > > > with
> > > > > > > > > the Kafka 4.0 upgrade is that it drops support for Kafka
> > > > > > > > > brokers
> > > > > > > > > running
> > > > > > > > > version 2.0.0 and lower. Given this, I think it would make
> > > > > > > > > sense to
> > > > > > > > > move
> > > > > > > > > the Connector version to 5.0.0 and maybe even move to Flink
> > > > > > > > > 2.1.0
> > > > > > > > > (which
> > > > > > > > > should be available in a month or so). This 5.0.0 release
> > > > > > > > > could also
> > > > > > > > > remove
> > > > > > > > > all the Zookeeper specific test infra and move to KRaft based
> > > > > > > > > clusters
> > > > > > > > > for
> > > > > > > > > testing. We could also move to a new, updated Flink Connector
> > > > > > > > > Parent
> > > > > > > > > pom
> > > > > > > > > version [4] which would harmonise the java versions and
> > > > > > > > > plugins with
> > > > > > > > > the
> > > > > > > > > main Flink project.
> > > > > > > > >
> > > > > > > > > I think, if the above is acceptable, that these changes
> > > > > > > > > warrant
> > > > > > > > > a major version bump. Users of older Kafka clusters would
> > > > > > > > > still be
> > > > > > > > > able to
> > > > > > > > > use 4.1.0 (which is an argument for making sure that release
> > > > > > > > > has the
> > > > > > > > > most
> > > > > > > > > up-to-date dependencies).
> > > > > > > > >
> > > > > > > > > Anyway, I would love to hear what the community think of the
> > > > > > > > > above.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Tom Cooper
> > > > > > > > > @tomcooper.dev | https://tomcooper.dev
> > > > > > > > >
> > > > > > > > > [0] https://github.com/apache/flink-connector-kafka/pull/180
> > > > > > > > > [1] https://nvd.nist.gov/vuln/detail/CVE-2025-27817
> > > > > > > > > [2] https://github.com/apache/flink-connector-kafka/pull/181
> > > > > > > > > [3] https://github.com/apache/flink-connector-kafka/pull/161
> > > > > > > > > [4]
> > > > > > > > > https://github.com/apache/flink-connector-shared-utils/pull/48
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road,
Winchester, Hampshire SO21 2JN
