FYI, I have posted a PR [1] for the Flink 2.1 update for the Kafka connector.
The main change with this update is that Flink has removed [2] (without deprecating first) the stripRowPrefix method from the DataTypeUtilsTest class. I couldn't see any evidence that this logic was moved to any other public API so this PR adds that logic into a util class within the connector. Also note that with the update to Flink 2.1.0 the Kafka connector drops support for Python 3.8, I added testing for Python 3.11 but 3.12-cython (Flink 2.1 added support for Python 3.12) is not yet available. If we can get his PR merged and the Kafka 4.0.0 update PR [3], we will be in a good position for the 5.0.0 release? Thanks, Tom Cooper @tomcooper.dev | https://tomcooper.dev [1] https://github.com/apache/flink-connector-kafka/pull/187 [2] https://github.com/apache/flink/pull/26784 On Thursday, 14 August 2025 at 21:59, Weiqing Yang <yangweiqing...@gmail.com> wrote: > Thanks, Fabian. This is helpful to know. > > On Thu, Aug 14, 2025 at 1:50 AM Fabian Paul fp...@apache.org wrote: > > > Hi Weiqing, > > > > So far, there is no concrete timeline to publish the Kafka 5.0 > > release. We are still releasing 4.0.1 at the moment. Regarding the > > Flink 2.1 support, you should be able to use the flink kafka connector > > that supports 2.0 also with 2.1. The underlying connector library is > > stable across minor versions. This also means that a new Flink minor > > release doesn't necessarily warrant a new connector release. > > > > Best, > > Fabian > > > > On Wed, Aug 13, 2025 at 6:57 PM Weiqing Yang yangweiqing...@gmail.com > > wrote: > > > > > Hi Tom, Fabian, > > > > > > Thanks for the updates on the v4.0.1 release. I noticed that v4.0.1 is > > > going out with Flink 2.0 (link > > > < > > > https://github.com/apache/flink-connector-kafka/blob/v4.0.1-rc2/pom.xml#L56 > > > ). > > > Do you have a timeline or target window in mind for the Flink Connector > > > Kafka 5.0 release that will include Flink 2.1 support? > > > > > > Thanks, > > > Weiqing > > > > > > On Mon, Aug 11, 2025 at 7:41 AM Tom Cooper c...@tomcooper.dev wrote: > > > > > > > Hi Fabian, > > > > > > > > Sorry, for the late reply, this message somehow ended up in my spam > > > > filter!? > > > > > > > > I think having the Flink 2.1 upgrade included in the move to Flink > > > > Connector Kafka 5.0 makes sense. > > > > I am hoping to find the time to work on the upgrade to Flink 2.1 at > > > > end of > > > > this week or next. > > > > Unless, of course, you are plan to work on that? > > > > > > > > Regards, > > > > > > > > Tom Cooper > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > On Tuesday, 29 July 2025 at 09:08, Fabian Paul > > > > fp...@confluent.io.INVALID > > > > wrote: > > > > > > > > > Hi Tom, > > > > > > > > > > Sounds good to me, I can start with the 4.0.1 release. > > > > > Regarding the 5.0 release, I am not super sure yet what to include. > > > > > Since releasing always takes some effort, I would also be okay with > > > > > doing the 5.0 release with incorporating Flink 2.1. The connector > > > > > already offers a release that is compatible with Flink 2.0, and in > > > > > theory, 2.1 should not introduce breaking changes that affect the > > > > > connector. > > > > > > > > > > Best, > > > > > Fabian > > > > > > > > > > On Mon, Jul 28, 2025 at 11:03 AM Tom Cooper c...@tomcooper.dev > > > > > wrote: > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > You make a good point, as there are only dependency updates, a > > > > > > 4.0.1 > > > > > > release makes more sense. > > > > > > > > > > > > At this point the 5.0 connector release could include the soon to > > > > > > be > > > > > > released Kafka 4.0.1 client libraries (the RC for that is out > > > > > > already). > > > > > > I assume we would want to leave the flink 2.1 upgrade to a future > > > > > > 5.1 > > > > > > release? > > > > > > > > > > > > Thanks for looking at this. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Tom Cooper > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > On Monday, 28 July 2025 at 09:51, Fabian Paul fp...@apache.org > > > > > > wrote: > > > > > > > > > > > > > Hi Tom, > > > > > > > > > > > > > > Thanks for starting this discussion. I think it's a good idea to > > > > > > > do > > > > > > > another 4.1.0 release before proceeding with 5.0 to offer a > > > > > > > release > > > > > > > with the vulnerability fixed without requiring users to upgrade > > > > > > > to > > > > > > > Kafka 4.0. Is there a reason you prefer to do the 4.1.0 release > > > > > > > instead of the 4.0.1 release? I reviewed the changes between the > > > > > > > current main and the release 4.0.0 [1], and they are mostly > > > > > > > dependency > > > > > > > upgrades and some fixes, but without any new features. What do > > > > > > > you > > > > > > > think about doing a 4.0.1 release and then kicking off 5.0.0 > > > > > > > with the > > > > > > > Kafka client upgrade? > > > > > > > > > > > > > > Best, > > > > > > > Fabian > > > > > > > > > > > > > > [1] > > > > > > > https://github.com/apache/flink-connector-kafka/compare/v4.0...main > > > > > > > > > > > > > > On Fri, Jul 25, 2025 at 11:58 AM Tom Cooper c...@tomcooper.dev > > > > > > > wrote: > > > > > > > > > > > > > > > Bumping this thread as we are now ready to merge the Kafka > > > > > > > > 4.0.0 > > > > > > > > client update PR [1]. This will bump the major version of the > > > > > > > > connector to > > > > > > > > 5.0, as we are dropping support for Kafka brokers running > > > > > > > > version > > > > > > > > 2.0.0 or > > > > > > > > earlier. > > > > > > > > > > > > > > > > However, I still think it would be worth doing a 4.1.0 release > > > > > > > > of > > > > > > > > the connector (with the Kafka 3.9.1 client), before the Kafka > > > > > > > > 4.0.0 > > > > > > > > client > > > > > > > > update is merged. > > > > > > > > > > > > > > > > The current Flink Kafka Connector (4.0) has a critical CVE [2], > > > > > > > > which is patched in the 3.9.1 Kafka client library (which the > > > > > > > > current > > > > > > > > main > > > > > > > > branch of the Flink connector is using). Doing a 4.1 release of > > > > > > > > the > > > > > > > > connector would cover any users of older Kafka versions that > > > > > > > > want this > > > > > > > > CVE > > > > > > > > patched and also give a stable release of the connector using a > > > > > > > > point > > > > > > > > release of the Kafka client (with all the bug fixes that > > > > > > > > entails). This > > > > > > > > would be a good option for users who don't want to jump > > > > > > > > straight onto > > > > > > > > the > > > > > > > > new major Kafka client version. > > > > > > > > > > > > > > > > What do people think? > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > [1] https://github.com/apache/flink-connector-kafka/pull/161 > > > > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2025-27817 > > > > > > > > > > > > > > > > On Wednesday, 9 July 2025 at 09:35, Tom Cooper > > > > > > > > c...@tomcooper.dev > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I would like to start a conversation about releases for the > > > > > > > > > Flink Connector Kafka project. > > > > > > > > > > > > > > > > > > We have recently updated [0] to version 3.9.1 of the Kafka > > > > > > > > > client library, which fixes a critical CVE [1]. With that in > > > > > > > > > mind, I > > > > > > > > > think > > > > > > > > > it would be prudent to have a 4.1.0 release as soon as > > > > > > > > > possible that > > > > > > > > > includes this. It would also be good to include the > > > > > > > > > dependency bumps > > > > > > > > > from > > > > > > > > > this PR [2] in that release. > > > > > > > > > > > > > > > > > > With the 4.1.0 release out, we could then move to looking at > > > > > > > > > the > > > > > > > > > Kafka 4.0 upgrade (there is already a PR [3] for that). The > > > > > > > > > main point > > > > > > > > > with > > > > > > > > > the Kafka 4.0 upgrade is that it drops support for Kafka > > > > > > > > > brokers > > > > > > > > > running > > > > > > > > > version 2.0.0 and lower. Given this, I think it would make > > > > > > > > > sense to > > > > > > > > > move > > > > > > > > > the Connector version to 5.0.0 and maybe even move to Flink > > > > > > > > > 2.1.0 > > > > > > > > > (which > > > > > > > > > should be available in a month or so). This 5.0.0 release > > > > > > > > > could also > > > > > > > > > remove > > > > > > > > > all the Zookeeper specific test infra and move to KRaft based > > > > > > > > > clusters > > > > > > > > > for > > > > > > > > > testing. We could also move to a new, updated Flink Connector > > > > > > > > > Parent > > > > > > > > > pom > > > > > > > > > version [4] which would harmonise the java versions and > > > > > > > > > plugins with > > > > > > > > > the > > > > > > > > > main Flink project. > > > > > > > > > > > > > > > > > > I think, if the above is acceptable, that these changes > > > > > > > > > warrant > > > > > > > > > a major version bump. Users of older Kafka clusters would > > > > > > > > > still be > > > > > > > > > able to > > > > > > > > > use 4.1.0 (which is an argument for making sure that release > > > > > > > > > has the > > > > > > > > > most > > > > > > > > > up-to-date dependencies). > > > > > > > > > > > > > > > > > > Anyway, I would love to hear what the community think of the > > > > > > > > > above. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Tom Cooper > > > > > > > > > @tomcooper.dev | https://tomcooper.dev > > > > > > > > > > > > > > > > > > [0] https://github.com/apache/flink-connector-kafka/pull/180 > > > > > > > > > [1] https://nvd.nist.gov/vuln/detail/CVE-2025-27817 > > > > > > > > > [2] https://github.com/apache/flink-connector-kafka/pull/181 > > > > > > > > > [3] https://github.com/apache/flink-connector-kafka/pull/161 > > > > > > > > > [4] > > > > > > > > > https://github.com/apache/flink-connector-shared-utils/pull/48
