Re: Kafka connector releases
Hi folks, thanks for the feedback. I'm still not convinced that it's the best approach to start with a bugfix release but I did a poor job in highlighting my thoughts. * I don't think we have any precedent of deprecating API in a bugfix release. * It's not obvious to me if a bugfix release counts towards our deprecation policy where we have one release between deprecation and removal. A user could use 3.2.0 not notice the 3.2.1 and see the deprecation only in 3.3.0. Then the user could be surprised by the removal in 4.0.0. (I'd personally see our policy too strict regarding major releases but that's what we did for Flink 2) * For the deprecation, I also introduced a new interface KafkaPartitioner that's a drop-in replacement for FlinkKafkaPartitioner. This is new API and should be part of a new minor release. * Finally, there is one new feature: [FLINK-22748] Allow dynamic target topic selection in SQL Kafka sinks [1]. * I'd drop 1.18 still and support 1.20 instead. * I'd also bump kafka-clients to 3.7.1 WDYT? Best, Arvid [1] https://github.com/apache/flink-connector-kafka/pull/109 On Sat, Sep 28, 2024 at 5:27 AM Leonard Xu wrote: > Thanks Arvid for the volunteering! > > + 1 for all of the three releases and RM candidate.Qingsheng and I would > like to help the 4.0.0-preview which follows the Flink 2.0 preview, > please feel free to ping us if you need any help. > > Btw, for other external connectors which highest supported flink version > still is 1.17 or 1.18, I also hope we can have dedicated plan to bump > their version ASAP, we can > start a new thread to track other external connector releases. > > Best, > Leonard > > > > 2024年9月27日 下午6:54，Qingsheng Ren 写道： > > > > Thanks for the work, Arvid! > > > > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If > > they are fully compatible, what about not dropping support for 1.18 in > > 3.2.1, and we release one more version 3.2.1-1.20? Then we can use > > 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18 > > and 1.19. > > > > And for the 4.0.0-preview version I'd like to help with it :-) > > > > Best, > > Qingsheng > > > > On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise wrote: > >> > >> Hi David, > >> > >> thank you very much for your reply. > >> > >>> Some thoughts on whether we need the 3 deliverables. And whether we > could > >> follow more traditional fixpack numbering: > >>> I see that there is already a release for 1.19 > >> > https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 > >> . > >> I am wondering why we need the first deliverable. If we need it for the > bug > >> fixes , why not have a 3.2.1? > >> I forgot the most important part of the first release: drop Flink 1.18 > >> support and add Flink 1.20 support. Hence I wouldn't want to mix that > into > >> a bugfix release. I think this is in line with the previous minor > releases. > >> > >>> I assume that kafka-3.4.0 will not work with previous Flink releases. > >> Would it be worth have a config switch to enable the lineage in the > >> connector so that we could use it with 1.19? We could maybe do a 3.3 if > >> this was the case. > >> Yes, as outlined in the discussion linked in the original message, we > need > >> to mixin new interfaces. Afaik classloading will fail if the interfaces > are > >> not present, even if the methods are not used. So I don't see how we can > >> use a config switch to make it happen (except with code duplication). > >> However, I'm grateful for any ideas to avoid this release. > >> > >> Best, > >> > >> Arvid > >> > >> On Fri, Sep 27, 2024 at 11:11 AM David Radley > >> wrote: > >> > >>> Hi Arvid, > >>> Some thoughts on whether we need the 3 deliverables. And whether we > could > >>> follow more traditional fixpack numbering: > >>> I see that there is already a release for 1.19 > >>> > https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 > >>> . I am wondering why we need the first deliverable. If we need it for > the > >>> bug fixes , why not have a 3.2.1? > >>> I assume that kafka-3.4.0 will not work with previous Flink releases. > >>> Would it be worth have a config switch to enable the lineage in the > >>> connector so that we could use it with 1.19? We could maybe do a 3.3 > if > >>> this was the case. > >>> > >>> WDYT? > >>> Kind regards, David. > >>> > >>> > >>> > >>> From: Arvid Heise > >>> Date: Friday, 27 September 2024 at 09:24 > >>> To: dev@flink.apache.org > >>> Subject: [EXTERNAL] Kafka connector releases > >>> Dear Flink devs, > >>> > >>> I'd like to initiate three(!) Kafka connector releases. The main > reason for > >>> having three releases is that we have been slacking a bit in keeping up > >>> with the latest changes. > >>> > >>> Here is the summary: > >>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap) > >>> - Incorporates lots of deprecations for Flink 2 including everything > that > >>> is related to FlinkKafkaProducer (SinkFunct
Re: Kafka connector releases
Thanks for driving this Arvid. +1 for the releases and the RM For the 4.0.0 version it would be good to drop the Flink version from the connector version (4.0.0, not 4.0.0-2.0). I started a discussion some time ago [1], however the thread went stale. But I think the feedback was generally aligned. I would propose we release 4.0.0 (or 4.0.0-preview?) and document this only supports Flink 2.0-preview. > I don't think we have any precedent of deprecating API in a bugfix release. I have not seen us do that before for connectors. I would personally prefer a 3.3.0 if we are dropping Flink 1.18 support, although I agree it is a grey area in the guidelines. This allows us to more easily create a 3.2.1-1.18 hotfix later incase the need arises and the community agrees to push another 1.18 hotfix. > It's not obvious to me if a bugfix release counts towards our deprecation policy where we have one release between deprecation and removal. Given that we encode the Flink version in the connector version, it is difficult/impossible to not notice it has been deprecated. Since the user would bump the version from 3.2.0-1.18 to 3.3.0-1.18, and this artifact would not be found in Maven central, and fail the build. > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If they are fully compatible, what about not dropping support for 1.18 in 3.2.1, and we release one more version 3.2.1-1.20? Then we can use 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18 and 1.19. This is a possibility, but still results in 3 releases, so only worth it if it is simpler Thanks, Danny [1] https://lists.apache.org/thread/byy8fgkr8dbrrdv2nxpxkrz18h904r9b On Mon, Sep 30, 2024 at 10:36 AM Arvid Heise wrote: > Hi folks, > > thanks for the feedback. I'm still not convinced that it's the best > approach to start with a bugfix release but I did a poor job in > highlighting my thoughts. > * I don't think we have any precedent of deprecating API in a bugfix > release. > * It's not obvious to me if a bugfix release counts towards our deprecation > policy where we have one release between deprecation and removal. A user > could use 3.2.0 not notice the 3.2.1 and see the deprecation only in 3.3.0. > Then the user could be surprised by the removal in 4.0.0. (I'd personally > see our policy too strict regarding major releases but that's what we did > for Flink 2) > * For the deprecation, I also introduced a new interface KafkaPartitioner > that's a drop-in replacement for FlinkKafkaPartitioner. This is new API and > should be part of a new minor release. > * Finally, there is one new feature: [FLINK-22748] Allow dynamic target > topic selection in SQL Kafka sinks [1]. > * I'd drop 1.18 still and support 1.20 instead. > * I'd also bump kafka-clients to 3.7.1 > > WDYT? > > Best, > > Arvid > > [1] https://github.com/apache/flink-connector-kafka/pull/109 > > On Sat, Sep 28, 2024 at 5:27 AM Leonard Xu wrote: > > > Thanks Arvid for the volunteering! > > > > + 1 for all of the three releases and RM candidate.Qingsheng and I would > > like to help the 4.0.0-preview which follows the Flink 2.0 preview, > > please feel free to ping us if you need any help. > > > > Btw, for other external connectors which highest supported flink version > > still is 1.17 or 1.18, I also hope we can have dedicated plan to bump > > their version ASAP, we can > > start a new thread to track other external connector releases. > > > > Best, > > Leonard > > > > > > > 2024年9月27日 下午6:54，Qingsheng Ren 写道： > > > > > > Thanks for the work, Arvid! > > > > > > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If > > > they are fully compatible, what about not dropping support for 1.18 in > > > 3.2.1, and we release one more version 3.2.1-1.20? Then we can use > > > 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18 > > > and 1.19. > > > > > > And for the 4.0.0-preview version I'd like to help with it :-) > > > > > > Best, > > > Qingsheng > > > > > > On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise wrote: > > >> > > >> Hi David, > > >> > > >> thank you very much for your reply. > > >> > > >>> Some thoughts on whether we need the 3 deliverables. And whether we > > could > > >> follow more traditional fixpack numbering: > > >>> I see that there is already a release for 1.19 > > >> > > > https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 > > >> . > > >> I am wondering why we need the first deliverable. If we need it for > the > > bug > > >> fixes , why not have a 3.2.1? > > >> I forgot the most important part of the first release: drop Flink 1.18 > > >> support and add Flink 1.20 support. Hence I wouldn't want to mix that > > into > > >> a bugfix release. I think this is in line with the previous minor > > releases. > > >> > > >>> I assume that kafka-3.4.0 will not work with previous Flink releases. > > >> Would it be worth have a config switch to enable the lineage in the > > >> connector s
[jira] [Created] (FLINK-36410) Improve Lineage Info Collection for flink app
Zhenqiu Huang created FLINK-36410: - Summary: Improve Lineage Info Collection for flink app Key: FLINK-36410 URL: https://issues.apache.org/jira/browse/FLINK-36410 Project: Flink Issue Type: Sub-task Affects Versions: 1.20.0 Reporter: Zhenqiu Huang -- This message was sent by Atlassian Jira (v8.20.10#820010)
FW: RE: Kafka connector releases
Hi again Arvid, I realise we actually have 3 releases (2 bug releases) supporting 1.18 – the renumbering threw me 😊. So I think your proposal looks good, kind regards, David. From: David Radley Date: Monday, 30 September 2024 at 17:07 To: dev@flink.apache.org Subject: [EXTERNAL] RE: Kafka connector releases Hi Arvid, Thanks for your reply on this. So we could do a 3.2.1, if we supported Flink 1.18? Is there a compelling reason to not support 1.18? I see that Kafka connectors that supported Flink 1.17 and below had 2 to 6 bug releases. Is there not an expectation that there will be a Flink 1.18 compatible Kafka connector bug release? If we end up having to do a 1.18 bug release due to demand and expectation, then maybe a 3.2.1 supporting 1.18 would make sense. It seems like we are changing the way we are doing the bug releases for the Kafka connector. Is the idea that there will be no bug release for 1.19 also? I assume 1.20 as the last v1 release would have bug releases. Thoughts? Kind regards, David. From: Arvid Heise Date: Friday, 27 September 2024 at 11:12 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Kafka connector releases Hi David, thank you very much for your reply. > Some thoughts on whether we need the 3 deliverables. And whether we could follow more traditional fixpack numbering: > I see that there is already a release for 1.19 https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 . I am wondering why we need the first deliverable. If we need it for the bug fixes , why not have a 3.2.1? I forgot the most important part of the first release: drop Flink 1.18 support and add Flink 1.20 support. Hence I wouldn't want to mix that into a bugfix release. I think this is in line with the previous minor releases. > I assume that kafka-3.4.0 will not work with previous Flink releases. Would it be worth have a config switch to enable the lineage in the connector so that we could use it with 1.19? We could maybe do a 3.3 if this was the case. Yes, as outlined in the discussion linked in the original message, we need to mixin new interfaces. Afaik classloading will fail if the interfaces are not present, even if the methods are not used. So I don't see how we can use a config switch to make it happen (except with code duplication). However, I'm grateful for any ideas to avoid this release. Best, Arvid On Fri, Sep 27, 2024 at 11:11 AM David Radley wrote: > Hi Arvid, > Some thoughts on whether we need the 3 deliverables. And whether we could > follow more traditional fixpack numbering: > I see that there is already a release for 1.19 > https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 > . I am wondering why we need the first deliverable. If we need it for the > bug fixes , why not have a 3.2.1? > I assume that kafka-3.4.0 will not work with previous Flink releases. > Would it be worth have a config switch to enable the lineage in the > connector so that we could use it with 1.19? We could maybe do a 3.3 if > this was the case. > > WDYT? >Kind regards, David. > > > > From: Arvid Heise > Date: Friday, 27 September 2024 at 09:24 > To: dev@flink.apache.org > Subject: [EXTERNAL] Kafka connector releases > Dear Flink devs, > > I'd like to initiate three(!) Kafka connector releases. The main reason for > having three releases is that we have been slacking a bit in keeping up > with the latest changes. > > Here is the summary: > 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap) > - Incorporates lots of deprecations for Flink 2 including everything that > is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer > (SourceFunction), and KafkaShuffle > - Lots of bugfixes that are very relevant for 1.19 users (and probably work > with older releases) > > 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later) > - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP > incorrectly says that it's avail in 1.19) > - We have discussed some alternatives to this release in [2] but basically > having a separate release is the cleanest solution. > - I'd like to linearize the releases to avoid having to do back or even > forward ports > > 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later) > - Much requested to get the connector out asap for the preview. (I think > the old jar using the removed interfaces should still work) > - Remove all deprecated things > - General spring cleaning (trying to get rid of arch unit violations, > migrate to JUnit5) > - Should we relocate the TableAPI stuff to o.a.f.connectors? > > I'd appreciate any feedback and volunteers for RM ;) If you have pending > PRs that should be part of any of those releases, please also write them. > > Best, > > Arvid > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener > [1] > > https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-23063828
Re: Kafka connector releases
Hi Danny, thanks for bringing this to my attention. I'm also in favor of starting with Flink 2.0. It's just a nice opportunity to break with some traditions and we are almost there anyways. We could resume the discussion over there if you'd like. Regarding the deprecation. I think you misunderstood me. I was talking about deprecating the KafkaShuffle and all the old API around Consumer/Producer (which we already deprecated in 1.14). [1] But I'd also deprecate 1.18 support with that release, so all that you say is additionally true. Since we are on the topic of deprecation and I got your attention: Does anyone have strong opinions about moving the TableAPI stuff into the o.a.f.connectors package? It would probably moving things + provide stubs with deprecation in the old location that extend from the actual things similar to the FlinkKafkaPartitioner in [1]. We can start a separate discussion and I don't have a strong opinion since I'm not a TableAPI/SQL guy. I'm also getting low on OSS contribution time. Best, Arvid [1] https://github.com/apache/flink-connector-kafka/pull/121 On Mon, Sep 30, 2024 at 4:49 PM Danny Cranmer wrote: > Thanks for driving this Arvid. > > +1 for the releases and the RM > > For the 4.0.0 version it would be good to drop the Flink version from the > connector version (4.0.0, not 4.0.0-2.0). I started a discussion some time > ago [1], however the thread went stale. But I think the feedback was > generally aligned. I would propose we release 4.0.0 (or 4.0.0-preview?) and > document this only supports Flink 2.0-preview. > > > I don't think we have any precedent of deprecating API in a bugfix > release. > > I have not seen us do that before for connectors. I would personally prefer > a 3.3.0 if we are dropping Flink 1.18 support, although I agree it is a > grey area in the guidelines. This allows us to more easily create a > 3.2.1-1.18 hotfix later incase the need arises and the community agrees to > push another 1.18 hotfix. > > > It's not obvious to me if a bugfix release counts towards our > deprecation policy where we have one release between deprecation and > removal. > > Given that we encode the Flink version in the connector version, it is > difficult/impossible to not notice it has been deprecated. Since the user > would bump the version from 3.2.0-1.18 to 3.3.0-1.18, and this artifact > would not be found in Maven central, and fail the build. > > > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If they > are fully compatible, what about not dropping support for 1.18 in 3.2.1, > and we release one more version 3.2.1-1.20? Then we can use 3.3.0 for the > new lineage feature in 1.20 and drop support for 1.18 and 1.19. > > This is a possibility, but still results in 3 releases, so only worth it if > it is simpler > > Thanks, > Danny > > [1] https://lists.apache.org/thread/byy8fgkr8dbrrdv2nxpxkrz18h904r9b > > > On Mon, Sep 30, 2024 at 10:36 AM Arvid Heise wrote: > > > Hi folks, > > > > thanks for the feedback. I'm still not convinced that it's the best > > approach to start with a bugfix release but I did a poor job in > > highlighting my thoughts. > > * I don't think we have any precedent of deprecating API in a bugfix > > release. > > * It's not obvious to me if a bugfix release counts towards our > deprecation > > policy where we have one release between deprecation and removal. A user > > could use 3.2.0 not notice the 3.2.1 and see the deprecation only in > 3.3.0. > > Then the user could be surprised by the removal in 4.0.0. (I'd personally > > see our policy too strict regarding major releases but that's what we did > > for Flink 2) > > * For the deprecation, I also introduced a new interface KafkaPartitioner > > that's a drop-in replacement for FlinkKafkaPartitioner. This is new API > and > > should be part of a new minor release. > > * Finally, there is one new feature: [FLINK-22748] Allow dynamic target > > topic selection in SQL Kafka sinks [1]. > > * I'd drop 1.18 still and support 1.20 instead. > > * I'd also bump kafka-clients to 3.7.1 > > > > WDYT? > > > > Best, > > > > Arvid > > > > [1] https://github.com/apache/flink-connector-kafka/pull/109 > > > > On Sat, Sep 28, 2024 at 5:27 AM Leonard Xu wrote: > > > > > Thanks Arvid for the volunteering! > > > > > > + 1 for all of the three releases and RM candidate.Qingsheng and I > would > > > like to help the 4.0.0-preview which follows the Flink 2.0 preview, > > > please feel free to ping us if you need any help. > > > > > > Btw, for other external connectors which highest supported flink > version > > > still is 1.17 or 1.18, I also hope we can have dedicated plan to bump > > > their version ASAP, we can > > > start a new thread to track other external connector releases. > > > > > > Best, > > > Leonard > > > > > > > > > > 2024年9月27日 下午6:54，Qingsheng Ren 写道： > > > > > > > > Thanks for the work, Arvid! > > > > > > > > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If > > >
Re: [VOTE] FLIP-477: Amazon SQS Source Connector
+1 (binding) On Fri, Sep 27, 2024 at 12:12 PM Danny Cranmer wrote: > +1 (binding) > > Thanks, > Danny > > On Fri, Sep 27, 2024 at 8:15 AM Arvid Heise wrote: > > > +1 (binding), > > > > Best, > > > > Arvid > > > > On Tue, Sep 3, 2024 at 12:35 PM Saurabh Singh < > saurabhsingh9...@gmail.com> > > wrote: > > > > > Hi Flink Devs, > > > > > > Gentle Reminder for voting on FLIP-477: Amazon SQS Source Connector > [1]. > > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-477 > > > +Amazon+SQS+Source+Connector > > > > > > Regards > > > Saurabh & Abhi > > > > > > On Thu, Aug 22, 2024 at 1:17 AM Aleksandr Pilipenko > > > wrote: > > > > > > > Thank you for driving this! > > > > +1 (non-binding) > > > > > > > > Best, > > > > Aleksandr > > > > > > > > On Wed, 21 Aug 2024 at 15:21, Ahmed Hamdy > > wrote: > > > > > > > > > Thanks for driving Abhisagar, > > > > > +1 (non-binding) > > > > > > > > > > Best Regards > > > > > Ahmed Hamdy > > > > > > > > > > > > > > > On Wed, 21 Aug 2024 at 09:11, Abhisagar Khatri < > > > > > khatri.abhisaga...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Flink Devs, > > > > > > > > > > > > Thank you for all the feedback about FLIP-477: Amazon SQS Source > > > > > Connector > > > > > > [1]. The discussion thread can be found here [2]. > > > > > > > > > > > > The vote will be open for at least 72 hours unless there are any > > > > > objections > > > > > > or insufficient votes. > > > > > > > > > > > > Regards, > > > > > > Abhi & Saurabh > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-477+Amazon+SQS+Source+Connector > > > > > > [2] > > https://lists.apache.org/thread/p27tj3kzyln1fjqyx2xmg4tt7thoh0sh > > > > > > > > > > > > > > > > > > > > >
[jira] [Created] (FLINK-36409) Publish some autoscaler metrics during stabilisation period
Gyula Fora created FLINK-36409: -- Summary: Publish some autoscaler metrics during stabilisation period Key: FLINK-36409 URL: https://issues.apache.org/jira/browse/FLINK-36409 Project: Flink Issue Type: Bug Components: Autoscaler, Kubernetes Operator Affects Versions: kubernetes-operator-1.9.0 Reporter: Gyula Fora Currently autoscaler metrics are not published at all during stabilisation period. This makes sense for most metrics as they rely on evaluation logic but for some such as PARALLELISM we should always publish them so users have metric continuity. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36413) Reenable the test case that related to Source/Sink APi deprecated.
LvYanquan created FLINK-36413: - Summary: Reenable the test case that related to Source/Sink APi deprecated. Key: FLINK-36413 URL: https://issues.apache.org/jira/browse/FLINK-36413 Project: Flink Issue Type: Technical Debt Components: Tests Affects Versions: 2.0-preview Reporter: LvYanquan Fix For: 2.0.0 In https://issues.apache.org/jira/browse/FLINK-36245 , we disabled some test cases because Kafka connector still used some removed class temporarily, After Kafka connector releases a new version that completed the deprecated work, we need to release these disabled test cases. 1. The temporary disabled test cases include: * SqlClientITCase#testMatchRecognize * connector test under flink-python/pyflink/datastream/conntectors/tests 2. Classes that skipped compile by maven plugin: * org.apache.flink.schema.registry.test.TestAvroConsumerConfluent * org.apache.flink.streaming.examples.statemachine.KafkaEventsGeneratorJob -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36414) NoSuchMethodError while trying flink cdc
Yang Hua Jie created FLINK-36414: Summary: NoSuchMethodError while trying flink cdc Key: FLINK-36414 URL: https://issues.apache.org/jira/browse/FLINK-36414 Project: Flink Issue Type: Bug Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: Yang Hua Jie flink: 1.20.0 flink-cdc: 3.2.0 mysql version: 8.0 kafka version: 2.2.1 cdc job ``` source: type: mysql hostname: localhost port: 3306 username: root password: 123456 tables: app_db.\.* server-time-zone: 'UTC' sink: type: kafka name: Kafka Sink properties.bootstrap.servers: PLAINTEXT://localhost:9092 pipeline: name: Sync MySQL Database to Kafka parallelism: 2 ``` Here is the error ``` 2024-10-01 10:05:14 java.lang.NoSuchMethodError: 'void org.apache.flink.formats.json.JsonRowDataSerializationSchema.(org.apache.flink.table.types.logical.RowType, org.apache.flink.formats.common.TimestampFormat, org.apache.flink.formats.json.JsonFormatOptions$MapNullKeyMode, java.lang.String, boolean)' at org.apache.flink.cdc.connectors.kafka.serialization.JsonSerializationSchema.buildSerializationForPrimaryKey(JsonSerializationSchema.java:134) at org.apache.flink.cdc.connectors.kafka.serialization.JsonSerializationSchema.serialize(JsonSerializationSchema.java:101) at org.apache.flink.cdc.connectors.kafka.serialization.JsonSerializationSchema.serialize(JsonSerializationSchema.java:47) at org.apache.flink.cdc.connectors.kafka.sink.PipelineKafkaRecordSerializationSchema.serialize(PipelineKafkaRecordSerializationSchema.java:99) at org.apache.flink.cdc.connectors.kafka.sink.PipelineKafkaRecordSerializationSchema.serialize(PipelineKafkaRecordSerializationSchema.java:44) at org.apache.flink.cdc.connectors.kafka.shaded.org.apache.flink.connector.kafka.sink.KafkaWriter.write(KafkaWriter.java:196) at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:160) at org.apache.flink.cdc.runtime.operators.sink.DataSinkWriterOperator.processElement(DataSinkWriterOperator.java:163) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38) at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) at java.base/java.lang.Thread.run(Thread.java:829) ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
RE: Kafka connector releases
Hi Arvid, Thanks for your reply on this. So we could do a 3.2.1, if we supported Flink 1.18? Is there a compelling reason to not support 1.18? I see that Kafka connectors that supported Flink 1.17 and below had 2 to 6 bug releases. Is there not an expectation that there will be a Flink 1.18 compatible Kafka connector bug release? If we end up having to do a 1.18 bug release due to demand and expectation, then maybe a 3.2.1 supporting 1.18 would make sense. It seems like we are changing the way we are doing the bug releases for the Kafka connector. Is the idea that there will be no bug release for 1.19 also? I assume 1.20 as the last v1 release would have bug releases. Thoughts? Kind regards, David. From: Arvid Heise Date: Friday, 27 September 2024 at 11:12 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Kafka connector releases Hi David, thank you very much for your reply. > Some thoughts on whether we need the 3 deliverables. And whether we could follow more traditional fixpack numbering: > I see that there is already a release for 1.19 https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 . I am wondering why we need the first deliverable. If we need it for the bug fixes , why not have a 3.2.1? I forgot the most important part of the first release: drop Flink 1.18 support and add Flink 1.20 support. Hence I wouldn't want to mix that into a bugfix release. I think this is in line with the previous minor releases. > I assume that kafka-3.4.0 will not work with previous Flink releases. Would it be worth have a config switch to enable the lineage in the connector so that we could use it with 1.19? We could maybe do a 3.3 if this was the case. Yes, as outlined in the discussion linked in the original message, we need to mixin new interfaces. Afaik classloading will fail if the interfaces are not present, even if the methods are not used. So I don't see how we can use a config switch to make it happen (except with code duplication). However, I'm grateful for any ideas to avoid this release. Best, Arvid On Fri, Sep 27, 2024 at 11:11 AM David Radley wrote: > Hi Arvid, > Some thoughts on whether we need the 3 deliverables. And whether we could > follow more traditional fixpack numbering: > I see that there is already a release for 1.19 > https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 > . I am wondering why we need the first deliverable. If we need it for the > bug fixes , why not have a 3.2.1? > I assume that kafka-3.4.0 will not work with previous Flink releases. > Would it be worth have a config switch to enable the lineage in the > connector so that we could use it with 1.19? We could maybe do a 3.3 if > this was the case. > > WDYT? >Kind regards, David. > > > > From: Arvid Heise > Date: Friday, 27 September 2024 at 09:24 > To: dev@flink.apache.org > Subject: [EXTERNAL] Kafka connector releases > Dear Flink devs, > > I'd like to initiate three(!) Kafka connector releases. The main reason for > having three releases is that we have been slacking a bit in keeping up > with the latest changes. > > Here is the summary: > 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap) > - Incorporates lots of deprecations for Flink 2 including everything that > is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer > (SourceFunction), and KafkaShuffle > - Lots of bugfixes that are very relevant for 1.19 users (and probably work > with older releases) > > 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later) > - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP > incorrectly says that it's avail in 1.19) > - We have discussed some alternatives to this release in [2] but basically > having a separate release is the cleanest solution. > - I'd like to linearize the releases to avoid having to do back or even > forward ports > > 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later) > - Much requested to get the connector out asap for the preview. (I think > the old jar using the removed interfaces should still work) > - Remove all deprecated things > - General spring cleaning (trying to get rid of arch unit violations, > migrate to JUnit5) > - Should we relocate the TableAPI stuff to o.a.f.connectors? > > I'd appreciate any feedback and volunteers for RM ;) If you have pending > PRs that should be part of any of those releases, please also write them. > > Best, > > Arvid > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener > [1] > > https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878 > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered
Re: Question around Flink's AdaptiveBatchScheduler
First of all my sincere apologies for slow progress on this one. @Junrui Lee and Xia, I updated the PR with respect to the last set of feedback comments on the PR. Please take a look. I am hoping to finish this one quickly. Regards Venkata krishnan On Thu, May 9, 2024 at 4:35 AM Venkatakrishnan Sowrirajan wrote: > Xia, > > Thanks for the reviews. Unfortunately due to work commitments I am little > delayed in addressing your review comments. Mostly will be done by end of > this week. Just a quick heads up. > > Jinrui, > > Thanks, that would be great. > > On Mon, May 6, 2024, 12:45 AM Junrui Lee wrote: > >> Hi, >> Thanks for the reminder. I will review it soon during my free time. >> >> Venkatakrishnan Sowrirajan 于2024年5月4日周六 10:10写道： >> >> > Jinrui and Xia >> > >> > Gentle ping for reviews. >> > >> > On Mon, Apr 29, 2024, 8:28 PM Venkatakrishnan Sowrirajan < >> vsowr...@asu.edu >> > > >> > wrote: >> > >> > > Hi Xia and Jinrui, >> > > >> > > Filed >> https://urldefense.com/v3/__https://github.com/apache/flink/pull/24736__;!!IKRxdwAv5BmarQ!YHVAC2UWD7ITI-Xyk6Flu6WhuSWYsHTLCOtJkLUhtIyojo0OxQOfsQmoS7d9-q0OhcRzbZ0hjO19qbictUrQLQ$ >> to address the above >> > > described issue. Please take a look whenever you can. >> > > >> > > Thanks >> > > Venkat >> > > >> > > >> > > On Thu, Apr 18, 2024 at 12:16 PM Venkatakrishnan Sowrirajan < >> > > vsowr...@asu.edu> wrote: >> > > >> > >> Filed >> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-35165__;!!IKRxdwAv5BmarQ!YHVAC2UWD7ITI-Xyk6Flu6WhuSWYsHTLCOtJkLUhtIyojo0OxQOfsQmoS7d9-q0OhcRzbZ0hjO19qbim4QLkBQ$ >> to address the >> > >> above described issue. Will share the PR here once it is ready for >> > review. >> > >> >> > >> Regards >> > >> Venkata krishnan >> > >> >> > >> >> > >> On Wed, Apr 17, 2024 at 5:32 AM Junrui Lee >> wrote: >> > >> >> > >>> Thanks Venkata and Xia for providing further clarification. I think >> > your >> > >>> example illustrates the significance of this proposal very well. >> Please >> > >>> feel free go ahead and address the concerns. >> > >>> >> > >>> Best, >> > >>> Junrui >> > >>> >> > >>> Venkatakrishnan Sowrirajan 于2024年4月16日周二 >> 07:01写道： >> > >>> >> > >>> > Thanks for adding your thoughts to this discussion. >> > >>> > >> > >>> > If we all agree that the source vertex parallelism shouldn't be >> bound >> > >>> by >> > >>> > the downstream max parallelism >> > >>> > (jobmanager.adaptive-batch-scheduler.max-parallelism) >> > >>> > based on the rationale and the issues described above, I can take >> a >> > >>> stab at >> > >>> > addressing the issue. >> > >>> > >> > >>> > Let me file a ticket to track this issue. Otherwise, I'm looking >> > >>> forward to >> > >>> > hearing more thoughts from others as well, especially Lijie and >> > Junrui >> > >>> who >> > >>> > have more context on the AdaptiveBatchScheduler. >> > >>> > >> > >>> > Regards >> > >>> > Venkata krishnan >> > >>> > >> > >>> > >> > >>> > On Mon, Apr 15, 2024 at 12:54 AM Xia Sun >> > wrote: >> > >>> > >> > >>> > > Hi Venkat, >> > >>> > > I agree that the parallelism of source vertex should not be >> upper >> > >>> bounded >> > >>> > > by the job's global max parallelism. The case you mentioned, >> >> > High >> > >>> > filter >> > >>> > > selectivity with huge amounts of data to read excellently >> supports >> > >>> this >> > >>> > > viewpoint. (In fact, in the current implementation, if the >> source >> > >>> > > parallelism is pre-specified at job create stage, rather than >> > >>> relying on >> > >>> > > the dynamic parallelism inference of the AdaptiveBatchScheduler, >> > the >> > >>> > source >> > >>> > > vertex's parallelism can indeed exceed the job's global max >> > >>> parallelism.) >> > >>> > > >> > >>> > > As Lijie and Junrui pointed out, the key issue is "semantic >> > >>> consistency." >> > >>> > > Currently, if a vertex has not set maxParallelism, the >> > >>> > > AdaptiveBatchScheduler will use >> > >>> > > `execution.batch.adaptive.auto-parallelism.max-parallelism` as >> the >> > >>> > vertex's >> > >>> > > maxParallelism. Since the current implementation does not >> > distinguish >> > >>> > > between source vertices and downstream vertices, source vertices >> > are >> > >>> also >> > >>> > > subject to this limitation. >> > >>> > > >> > >>> > > Therefore, I believe that if the issue of "semantic consistency" >> > can >> > >>> be >> > >>> > > well explained in the code and configuration documentation, the >> > >>> > > AdaptiveBatchScheduler should support that the parallelism of >> > source >> > >>> > > vertices can exceed the job's global max parallelism. >> > >>> > > >> > >>> > > Best, >> > >>> > > Xia >> > >>> > > >> > >>> > > Venkatakrishnan Sowrirajan 于2024年4月14日周日 >> > 10:31写道： >> > >>> > > >> > >>> > > > Let me state why I think "*jobmanager.adaptive-batch-sche* >> > >>> > > > *duler.default-source-parallelism*" should not be bound by >> the " >> > >>> > > > *jobmanager.adaptive-batch-sche**duler.max-parallelism*". >> > >>>
[jira] [Created] (FLINK-36411) Allow configuring job vertex throughput limits for auto scaler
Sai Sharath Dandi created FLINK-36411: - Summary: Allow configuring job vertex throughput limits for auto scaler Key: FLINK-36411 URL: https://issues.apache.org/jira/browse/FLINK-36411 Project: Flink Issue Type: Improvement Components: Autoscaler Reporter: Sai Sharath Dandi *Problem Statement* Currently, auto scaler has the ability to detect ineffective scalings and prevent further scale ups. However, the ineffective scaling detection does not work when there is no scaling history. Moreover, the ineffective scaling detection does not prevent the job from making an ineffective scaling for the first time. This is particularly important for some of the sinks that could enforce throttling (For example, quota limit for Kafka sink). In these cases, we can avoid ineffective scalings by comparing the throughput limit and current processing rate. *Solution* Some high level ideas below # Allow user to specify the throughput limit at job vertex level -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36412) Retry strategies on Kinesis Sink
Ed Berezitsky created FLINK-36412:
-
Summary: Retry strategies on Kinesis Sink
Key: FLINK-36412
URL: https://issues.apache.org/jira/browse/FLINK-36412
Project: Flink
Issue Type: Improvement
Components: Connectors / AWS, Connectors / Kinesis
Reporter: Ed Berezitsky
In case of internal Kinesis issues, Kinesis Sink connector crashes and causes
application to stop
{{java.util.concurrent.CompletionException:
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.exception.SdkClientException:
Unable to execute HTTP request: connection timed
out:.data-kinesis.us-west-2.amazonaws.com/<10.x.x.x>:443 at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)}}
The connector could introduce improved exception/error managing mechanism to
allow an application to wait until connection is restored, controlled by
additional parameters available for a users of the connector to set in a Flink
aopp
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
