Proposal to Add a JDBC Sink Plugin to Flink-CDC-Pipeline
Hi all My name is ZhengjunZhou, an user and developer of FlinkCDC. In my recent projects, I realized that we could enhance the capabilities of Flink-CDC-Pipeline by introducing a JDBC Sink plugin, enabling FlinkCDC to directly output change data capture (CDC) to various JDBC-supported database systems. Currently, while FlinkCDC offers support for a wide range of data sources, there is no direct solution for sinks, especially for common relational databases. I believe that adding a JDBC Sink plugin will significantly boost its applicability in data integration scenarios. Specifically, this plugin would allow users to configure database connections and stream data directly to SQL databases via the standard JDBC interface. This could be used for data migration tasks as well as real-time data synchronization. To further discuss this proposal and gather feedback from the community, I have prepared a preliminary design draft and hope to discuss it in detail in the upcoming community meeting. Please consider the potential value of this feature and provide your insights and guidance. Thank you for your time and consideration. I look forward to your active feedback and further discussion. [1] https://github.com/apache/flink-connector-jdbc
[DISCUSS] Add a JDBC Sink Plugin to Flink-CDC-Pipeline
Hi all My name is ZhengjunZhou, an user and developer of FlinkCDC. In my recent projects, I realized that we could enhance the capabilities of Flink-CDC-Pipeline by introducing a JDBC Sink plugin, enabling FlinkCDC to directly output change data capture (CDC) to various JDBC-supported database systems. Currently, while FlinkCDC offers support for a wide range of data sources, there is no direct solution for sinks, especially for common relational databases. I believe that adding a JDBC Sink plugin will significantly boost its applicability in data integration scenarios. Specifically, this plugin would allow users to configure database connections and stream data directly to SQL databases via the standard JDBC interface. This could be used for data migration tasks as well as real-time data synchronization. To further discuss this proposal and gather feedback from the community, I have prepared a preliminary design draft and hope to discuss it in detail in the upcoming community meeting. Please consider the potential value of this feature and provide your insights and guidance. Thank you for your time and consideration. I look forward to your active feedback and further discussion. [1] https://github.com/apache/flink-connector-jdbc
Re: [DISCUSS] FLIP suggestion: Flink SQL Scalar Functions.
Hello dev! Previously I send the FLIP[1] suggestion about creating scalar funcitons in SQL syntax and still want to discuss expediency of creating scalar (may be in future anothers functions like TableFunction) functions from SQL. Something like this already maked by DataBricks[2] in AWS Spark but in strict limits. My suggestion is creating UDF ScalarFunction via SQL, let's discuss and will be great if someone suggest vision of this feature. [1] https://docs.google.com/document/d/1Os0aRLAXYxmcO-GkaAfxlZgi3kipntA28tFoA4kdzno/edit?usp=sharing [2] https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html#create-and-use-a-sql-scalar-function >Вторник, 13 февраля 2024, 16:12 +07:00 от Сергей Парышев >: > >Accroding this thread >https://lists.apache.org/thread/rkpvlnwj9gv1hvx1dyklx6k88qpnvk2t , I want to >suggest FLIP via Google Doc, >https://docs.google.com/document/d/1Os0aRLAXYxmcO-GkaAfxlZgi3kipntA28tFoA4kdzno/edit?usp=sharing > >
Re: [DISCUSS] FLIP-XXX: Improve JDBC connector extensibility for Table API
Hello Muhammet and Jeyhun! Thanks for your comments! @Jeyhun: > Could you please elaborate more on how the new approach will be backwards compatible? In the FLIP I provide how the current Factories in JDBC would be changed with this refactor, do you mean something different? Can you be more specific with your request? On May 14, 2024 at 12:32 +0200, Jeyhun Karimov , wrote: > Hi Lorenzo, > > Thanks for driving this FLIP. +1 for it. > > Could you please elaborate more on how the new approach will be backwards > compatible? > > Regards, > Jeyhun > > On Tue, May 14, 2024 at 10:00 AM Muhammet Orazov > wrote: > > > Hey Lorenzo, > > > > Thanks for driving this FLIP! +1 > > > > It will improve the user experience of using JDBC based > > connectors and help developers to build with different drivers. > > > > Best, > > Muhammet > > > > On 2024-05-13 10:20, lorenzo.affe...@ververica.com.INVALID wrote: > > > > Hello dev! > > > > > > > > I want to share a draft of my FLIP to refactor the JDBC connector to > > > > improve its extensibility [1]. > > > > The goal is to allow implementers to write new connectors on top of the > > > > JDBC one for Table API with clean and maintainable code. > > > > > > > > Any feedback from the community is more and welcome. > > > > > > > > [1] > > > > > > https://docs.google.com/document/d/1kl_AikMlqPUI-LNiPBraAFVZDRg1LF4bn6uiNtR4dlY/edit?usp=sharing > >
[jira] [Created] (FLINK-35358) Breaking change when loading artifacts
Rasmus Thygesen created FLINK-35358:
---
Summary: Breaking change when loading artifacts
Key: FLINK-35358
URL: https://issues.apache.org/jira/browse/FLINK-35358
Project: Flink
Issue Type: New Feature
Components: Client / Job Submission, flink-docker
Affects Versions: 1.19.0
Reporter: Rasmus Thygesen
We have been using the following code snippet in our Dockerfiles for running a
Flink job in application mode
{code:java}
FROM flink:1.18.1-scala_2.12-java17
COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/artifacts/my-job.jar
USER flink {code}
Which has been working since at least around Flink 1.14, but the 1.19 update
has broken our Dockerfiles. The fix is to put the jar file a step further out
so the code snippet becomes
{code:java}
FROM flink:1.18.1-scala_2.12-java17
COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
USER flink {code}
We have not spent too much time looking into what the cause is, but we get the
stack trace
{code:java}
myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load the
provided entrypoint class.myjob-jobmanager-1 | at
org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | Caused by:
org.apache.flink.client.program.ProgramInvocationException: The program's entry
point class 'my.company.job.MyJob' was not found in the jar
file.myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | ... 4
moremyjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
my.company.job.MyJobmyjob-jobmanager-1 | at
java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]myjob-jobmanager-1 |
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]myjob-jobmanager-1
| at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]myjob-jobmanager-1 |
at
org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
java.lang.Class.forName0(Native Method) ~[?:?]myjob-jobmanager-1 | at
java.lang.Class.forName(Unknown Source) ~[?:?]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:479)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
~[flink-dist-1.19.0.jar:1.19.0]myjob-jobmanager-1 | at
Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #3
+1 (non-binding) - Tag is present - Verified checksums and hashes - Verified signatures - Verified no binaries in source - Build successful with jdk8, jdk11 - Checked the dist jar was built by jdk8 Best, Jiabao Leonard Xu 于2024年5月14日周二 19:53写道： > +1 (binding) > > - verified signatures > - verified hashsums > - checked Github release tag > - checked release notes > - Run pipeline from MySQL to StarRocks with fields projection, the result > is expected > - Run pipeline from MySQL to StarRocks with filter, the result is expected > - reviewed Jira issues for cdc-3.1.0，and closed invalid issue for > incorrect version 3.1.0 > - reviewed the web PR and left two minor comments > > Best, > Leonard > > > 2024年5月14日 下午6:07，Yanquan Lv 写道： > > > > +1 (non-binding) > > - Validated checksum hash > > - Build the source with Maven and jdk8 > > - Verified web PR > > - Check that the jar is built by jdk8 > > - Check synchronizing from mysql to paimon > > - Check synchronizing from mysql to kafka > > > > Hang Ruan 于2024年5月13日周一 13:55写道： > > > >> +1 (non-binding) > >> > >> - Validated checksum hash > >> - Verified signature > >> - Verified that no binaries exist in the source archive > >> - Build the source with Maven and jdk8 > >> - Verified web PR > >> - Check that the jar is built by jdk8 > >> - Check synchronizing schemas and data from mysql to starrocks following > >> the quickstart > >> > >> Best, > >> Hang > >> > >> Qingsheng Ren 于2024年5月11日周六 10:10写道： > >> > >>> Hi everyone, > >>> > >>> Please review and vote on the release candidate #3 for the version > 3.1.0 > >> of > >>> Apache Flink CDC, as follows: > >>> [ ] +1, Approve the release > >>> [ ] -1, Do not approve the release (please provide specific comments) > >>> > >>> **Release Overview** > >>> > >>> As an overview, the release consists of the following: > >>> a) Flink CDC source release to be deployed to dist.apache.org > >>> b) Maven artifacts to be deployed to the Maven Central Repository > >>> > >>> **Staging Areas to Review** > >>> > >>> The staging areas containing the above mentioned artifacts are as > >> follows, > >>> for your review: > >>> * All artifacts for a) can be found in the corresponding dev repository > >> at > >>> dist.apache.org [1], which are signed with the key with fingerprint > >>> A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B [2] > >>> * All artifacts for b) can be found at the Apache Nexus Repository [3] > >>> > >>> Other links for your review: > >>> * JIRA release notes [4] > >>> * Source code tag "release-3.1.0-rc3" with commit hash > >>> 5452f30b704942d0ede64ff3d4c8699d39c63863 [5] > >>> * PR for release announcement blog post of Flink CDC 3.1.0 in flink-web > >> [6] > >>> > >>> **Vote Duration** > >>> > >>> The voting time will run for at least 72 hours, adopted by majority > >>> approval with at least 3 PMC affirmative votes. > >>> > >>> Thanks, > >>> Qingsheng Ren > >>> > >>> [1] https://dist.apache.org/repos/dist/dev/flink/flink-cdc-3.1.0-rc3/ > >>> [2] https://dist.apache.org/repos/dist/release/flink/KEYS > >>> [3] > >> https://repository.apache.org/content/repositories/orgapacheflink-1733 > >>> [4] > >>> > >>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354387 > >>> [5] https://github.com/apache/flink-cdc/releases/tag/release-3.1.0-rc3 > >>> [6] https://github.com/apache/flink-web/pull/739 > >>> > >> > >
Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #3
+1 (non-binding) - Verified signature and checksum hash - Verified that no binaries exist in the source archive - Build source code successful on ubuntu 22.04 with jdk8 - Check tag and branch exist - Check jars are built by jdk8 Best, Zhongqiang Gong Qingsheng Ren 于2024年5月11日周六 10:10写道： > Hi everyone, > > Please review and vote on the release candidate #3 for the version 3.1.0 of > Apache Flink CDC, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > **Release Overview** > > As an overview, the release consists of the following: > a) Flink CDC source release to be deployed to dist.apache.org > b) Maven artifacts to be deployed to the Maven Central Repository > > **Staging Areas to Review** > > The staging areas containing the above mentioned artifacts are as follows, > for your review: > * All artifacts for a) can be found in the corresponding dev repository at > dist.apache.org [1], which are signed with the key with fingerprint > A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B [2] > * All artifacts for b) can be found at the Apache Nexus Repository [3] > > Other links for your review: > * JIRA release notes [4] > * Source code tag "release-3.1.0-rc3" with commit hash > 5452f30b704942d0ede64ff3d4c8699d39c63863 [5] > * PR for release announcement blog post of Flink CDC 3.1.0 in flink-web [6] > > **Vote Duration** > > The voting time will run for at least 72 hours, adopted by majority > approval with at least 3 PMC affirmative votes. > > Thanks, > Qingsheng Ren > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-cdc-3.1.0-rc3/ > [2] https://dist.apache.org/repos/dist/release/flink/KEYS > [3] https://repository.apache.org/content/repositories/orgapacheflink-1733 > [4] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354387 > [5] https://github.com/apache/flink-cdc/releases/tag/release-3.1.0-rc3 > [6] https://github.com/apache/flink-web/pull/739 >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Thank you Martijn for the detailed FLIP. Will save this for the future as an example for these kind of FLIPs :) This also shakes the adoption of the new interface for downstream connectors. Volunteered for the pubsub port (table API), and trying to assess the current status of pending PRs. I give you my +1 (non-bingind) :) On May 13, 2024 at 11:52 +0200, Ahmed Hamdy , wrote: > Thanks for the clarification. > +1 to starting the vote. > Best Regards > Ahmed Hamdy > > > On Mon, 13 May 2024 at 09:10, Martijn Visser > wrote: > > > Hi Ahmed, > > > > There's no reason to refrain from releases for Flink 1.* versions: these > > connector implementations are still supported in the Flink 1.* series. > > > > Best regards, > > > > Martijn > > > > On Sun, May 12, 2024 at 5:55 PM Ahmed Hamdy wrote: > > > > > > Thanks Martijn > > > > I believe you missed my question, > > > > > > > > Should this change take place in 1.20, what are the planned release > > > > steps > > > > > > for connectors that only offer a deprecated interface in this case > > (i.e. > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from > > > > releases > > > > > > till the blockers are implemented? > > > > > > > > > > > > > > Could you please clarify? > > > > > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Sun, 12 May 2024 at 14:07, Martijn Visser > > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > > > If there are no more considerations, I'll open up a vote in the next > > > > couple > > > > > > of days. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > On Wed, May 8, 2024 at 4:08 AM Hongshun Wang > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Martijn, Thanks for the proposal +1 from me.Some sinks still > > > > > > > > use > > > > > > > > sinkfunction; it's time to take a step forward. > > > > > > > > > > > > > > > > Best, > > > > > > > > Hongshun > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > +1 from my side, thanks Martijn for the effort. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy 写道： > > > > > > > > > > > > > > > > > > > > > > > > Hi Martijn > > > > > > > > > > > > Thanks for the proposal +1 from me. > > > > > > > > > > > > Should this change take place in 1.20, what are the > > > > > > > > > > > > planned > > release > > > > > > > > steps > > > > > > > > > > > > for connectors that only offer a deprecated interface > > > > > > > > > > > > in this > > case > > > > > > > > (i.e. > > > > > > > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to > > > > > > > > > > > > refrain from > > > > > > > > > > releases > > > > > > > > > > > > that support 1.20+ till the blockers are implemented? > > > > > > > > > > > > Best Regards > > > > > > > > > > > > Ahmed Hamdy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 3 May 2024 at 14:32, Péter Váry < > > > > peter.vary.apa...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >>> With regards to FLINK-35149, the fix version > > > > > > > > > > > > > >>> indicates a change > > > > at > > > > > > > > > > Flink > > > > > > > > > > > > >> CDC; is that indeed correct, or does it require a > > > > > > > > > > > > >> change in the > > > > > > SinkV2 > > > > > > > > > > > > >> interface? > > > > > > > > > > > > >> > > > > > > > > > > > > >> The fix doesn't need change in SinkV2, so we are > > > > > > > > > > > > >> good there. > > > > > > > > > > > > >> The issue is that the new SinkV2 > > > > > > > > > > SupportsCommitter/SupportsPreWriteTopology > > > > > > > > > > > > >> doesn't work with the CDC yet. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Martijn Visser ezt írta > > > > > > > > > > > > >> (időpont: > > > > 2024. > > > > > > > > máj. > > > > > > > > > > > > >> 3., > > > > > > > > > > > > >> P, 14:06): > > > > > > > > > > > > >> > > > > > > > > > > > > > >>> Hi Ferenc, > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> You're right, 1.20 it is :) > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> I've assigned the HBase one to you! > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> Thanks, > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> Martijn > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Hi Martijn, > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 for the proposal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > targeted for Flink 1.19 > > > > > > > > > > > > > > > > > > > > > > > > > > > > I guess you meant Fl
[VOTE] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi everyone, Thanks for all the feedback about the FLIP-450: Improve Runtime Configuration for Flink 2.0 [1] [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours(excluding weekends,until MAY 20, 12:00AM GMT) unless there is an objection or an insufficient number of votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 [2] https://lists.apache.org/thread/20mkzd31607posls793hxy7mht40xp2x Best regards, Xuannan
Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
As said on the discussion thread, +1 (non-binding) Thank you! On May 14, 2024 at 23:03 +0200, Rodrigo Meneses , wrote: > +1 (non-binding) > Thanks a lot for driving this effort!! > -Rodrigo > > > On Tue, May 14, 2024 at 11:00 AM Péter Váry > wrote: > > > +1 (non-binding) > > Thanks, Peter > > > > On Tue, May 14, 2024, 09:50 gongzhongqiang > > wrote: > > > > > > +1 (non-binding) > > > > > > > > Best. > > > > Zhongqiang Gong > > > > > > > > Martijn Visser 于2024年5月14日周二 14:45写道： > > > > > > > > > > Hi everyone, > > > > > > > > > > > > With no more discussions being open in the thread [1] I would like > > > > > > to > > > > start > > > > > > a vote on FLIP-453: Promote Unified Sink API V2 to Public and > > > > > > Deprecate > > > > > > SinkFunction [2] > > > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > objection > > > > or > > > > > > insufficient votes. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59 > > > > > > [2] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction > > > > > > > > > > > >
Re: [VOTE] FLIP-450: Improve Runtime Configuration for Flink 2.0
+1(binding) Best, Rui On Wed, May 15, 2024 at 5:01 PM Xuannan Su wrote: > Hi everyone, > > Thanks for all the feedback about the FLIP-450: Improve Runtime > Configuration for Flink 2.0 [1] [2]. > > I'd like to start a vote for it. The vote will be open for at least 72 > hours(excluding weekends,until MAY 20, 12:00AM GMT) unless there is an > objection or an insufficient number of votes. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > [2] https://lists.apache.org/thread/20mkzd31607posls793hxy7mht40xp2x > > > Best regards, > Xuannan >
[jira] [Created] (FLINK-35359) General Improvement to Configuration for Flink 2.0
Xuannan Su created FLINK-35359: -- Summary: General Improvement to Configuration for Flink 2.0 Key: FLINK-35359 URL: https://issues.apache.org/jira/browse/FLINK-35359 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xuannan Su As Flink moves toward version 2.0, we want to provide users with a better experience with the existing configuration. In this FLIP, we outline several general improvements to the current configuration: * Ensure all the ConfigOptions are properly annotated * Ensure all user-facing configurations are included in the documentation generation process * Make the existing ConfigOptions use the proper type * Mark all internally used ConfigOptions with the @Internal annotation https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35360) [Feature] Submit Flink CDC pipeline job to yarn Application mode
王俊博 created FLINK-35360: --- Summary: [Feature] Submit Flink CDC pipeline job to yarn Application mode Key: FLINK-35360 URL: https://issues.apache.org/jira/browse/FLINK-35360 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: 王俊博 For now flink-cdc pipeline support cli yarn session mode submit.I'm willing to support yarn application mode submit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[DISCUSSION] FLIP-457: Improve Table/SQL Configuration for Flink 2.0
Hi all, I'd like to start a discussion on FLIP-457: Improve Table/SQL Configuration for Flink 2.0 [1]. This FLIP revisited all Table/SQL configurations to improve user-friendliness and maintainability as Flink moves toward 2.0. I am looking forward to your feedback. Best regards, Jane [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=307136992
Re: [VOTE] FLIP-450: Improve Runtime Configuration for Flink 2.0
+1(binding) Best regards, Weijie Rui Fan <1996fan...@gmail.com> 于2024年5月15日周三 17:50写道： > +1(binding) > > Best, > Rui > > On Wed, May 15, 2024 at 5:01 PM Xuannan Su wrote: > > > Hi everyone, > > > > Thanks for all the feedback about the FLIP-450: Improve Runtime > > Configuration for Flink 2.0 [1] [2]. > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > hours(excluding weekends,until MAY 20, 12:00AM GMT) unless there is an > > objection or an insufficient number of votes. > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > [2] https://lists.apache.org/thread/20mkzd31607posls793hxy7mht40xp2x > > > > > > Best regards, > > Xuannan > > >
Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Thanks Martijn driving this! +1 (non-binding) Best, Muhammet On 2024-05-14 06:43, Martijn Visser wrote: Hi everyone, With no more discussions being open in the thread [1] I would like to start a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction [2] The vote will be open for at least 72 hours unless there is an objection or insufficient votes. Best regards, Martijn [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59 [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction
Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
+1(binding) Thanks Martijn! Best regards, Jing On Wed, May 15, 2024 at 7:00 PM Muhammet Orazov wrote: > Thanks Martijn driving this! +1 (non-binding) > > Best, > Muhammet > > On 2024-05-14 06:43, Martijn Visser wrote: > > Hi everyone, > > > > With no more discussions being open in the thread [1] I would like to > > start > > a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate > > SinkFunction [2] > > > > The vote will be open for at least 72 hours unless there is an > > objection or > > insufficient votes. > > > > Best regards, > > > > Martijn > > > > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59 > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction >
[jira] [Created] (FLINK-35361) Delete Flinkhistory files that failed to write to the local directory
dengxaing created FLINK-35361: - Summary: Delete Flinkhistory files that failed to write to the local directory Key: FLINK-35361 URL: https://issues.apache.org/jira/browse/FLINK-35361 Project: Flink Issue Type: Bug Reporter: dengxaing Attachments: image-2024-05-15-21-15-54-973.png I found a bug in org.apache.flink.runtime.webmonitor.history.HistoryServerStaticFileServerHandler !image-2024-05-15-21-15-54-973.png! When the local directory is full, the above code will create an empty or incomplete file in local. At this point, the Flink History webui page cannot be open or display abnormally. However, when the local directory is expanded, the Flink History webui page will not return to normal because new files will not be regenerated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
+1 (non-binding). Thanks Martijn! Best Regards Ahmed Hamdy On Wed, 15 May 2024 at 14:11, Jing Ge wrote: > +1(binding) Thanks Martijn! > > Best regards, > Jing > > On Wed, May 15, 2024 at 7:00 PM Muhammet Orazov > wrote: > > > Thanks Martijn driving this! +1 (non-binding) > > > > Best, > > Muhammet > > > > On 2024-05-14 06:43, Martijn Visser wrote: > > > Hi everyone, > > > > > > With no more discussions being open in the thread [1] I would like to > > > start > > > a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate > > > SinkFunction [2] > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection or > > > insufficient votes. > > > > > > Best regards, > > > > > > Martijn > > > > > > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59 > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction > > >
Re: [DISCUSS] FLIP-XXX: Improve JDBC connector extensibility for Table API
Hi Lorenzo, This seems like a very useful addition. +1 (non-binding) from my side. I echo Jeyhun's question about backward compatibility as it is not mentioned in the FLIP. Best Regards Ahmed Hamdy On Wed, 15 May 2024 at 08:12, wrote: > Hello Muhammet and Jeyhun! > Thanks for your comments! > > @Jeyhun: > > > Could you please elaborate more on how the new approach will be backwards > compatible? > > In the FLIP I provide how the current Factories in JDBC would be changed > with this refactor, do you mean something different? Can you be more > specific with your request? > On May 14, 2024 at 12:32 +0200, Jeyhun Karimov , > wrote: > > Hi Lorenzo, > > > > Thanks for driving this FLIP. +1 for it. > > > > Could you please elaborate more on how the new approach will be backwards > > compatible? > > > > Regards, > > Jeyhun > > > > On Tue, May 14, 2024 at 10:00 AM Muhammet Orazov > > wrote: > > > > > Hey Lorenzo, > > > > > > Thanks for driving this FLIP! +1 > > > > > > It will improve the user experience of using JDBC based > > > connectors and help developers to build with different drivers. > > > > > > Best, > > > Muhammet > > > > > > On 2024-05-13 10:20, lorenzo.affe...@ververica.com.INVALID wrote: > > > > > Hello dev! > > > > > > > > > > I want to share a draft of my FLIP to refactor the JDBC connector > to > > > > > improve its extensibility [1]. > > > > > The goal is to allow implementers to write new connectors on top > of the > > > > > JDBC one for Table API with clean and maintainable code. > > > > > > > > > > Any feedback from the community is more and welcome. > > > > > > > > > > [1] > > > > > > > > > https://docs.google.com/document/d/1kl_AikMlqPUI-LNiPBraAFVZDRg1LF4bn6uiNtR4dlY/edit?usp=sharing > > > >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Hi Lorenzo, yes it is pretty useful to start tackling the ones that still don't implement the new interface, happy so sync offline to close this task, additionally a volunteer to review the migrations would be highly appreciated. Best Regards Ahmed Hamdy On Wed, 15 May 2024 at 10:01, wrote: > Thank you Martijn for the detailed FLIP. > Will save this for the future as an example for these kind of FLIPs :) > > This also shakes the adoption of the new interface for downstream > connectors. > Volunteered for the pubsub port (table API), and trying to assess the > current status of pending PRs. > > I give you my +1 (non-bingind) :) > On May 13, 2024 at 11:52 +0200, Ahmed Hamdy , wrote: > > Thanks for the clarification. > > +1 to starting the vote. > > Best Regards > > Ahmed Hamdy > > > > > > On Mon, 13 May 2024 at 09:10, Martijn Visser > > wrote: > > > > > Hi Ahmed, > > > > > > There's no reason to refrain from releases for Flink 1.* versions: > these > > > connector implementations are still supported in the Flink 1.* series. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Sun, May 12, 2024 at 5:55 PM Ahmed Hamdy > wrote: > > > > > > > > Thanks Martijn > > > > > I believe you missed my question, > > > > > > > > > > Should this change take place in 1.20, what are the planned > release steps > > > > > > > for connectors that only offer a deprecated interface in this > case > > > (i.e. > > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain > from > > > > > releases > > > > > > > till the blockers are implemented? > > > > > > > > > > > > > > > > > Could you please clarify? > > > > > > > > > > Best Regards > > > > > Ahmed Hamdy > > > > > > > > > > > > > > > On Sun, 12 May 2024 at 14:07, Martijn Visser < > martijnvis...@apache.org> > > > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > If there are no more considerations, I'll open up a vote in > the next > > > > > couple > > > > > > > of days. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Martijn > > > > > > > > > > > > > > On Wed, May 8, 2024 at 4:08 AM Hongshun Wang < > loserwang1...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Martijn, Thanks for the proposal +1 from me.Some sinks > still use > > > > > > > > > sinkfunction; it's time to take a step forward. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Hongshun > > > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu < > xbjt...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > +1 from my side, thanks Martijn for the effort. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > > > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy > 写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Martijn > > > > > > > > > > > > > Thanks for the proposal +1 from me. > > > > > > > > > > > > > Should this change take place in 1.20, what are > the planned > > > release > > > > > > > > > steps > > > > > > > > > > > > > for connectors that only offer a deprecated > interface in this > > > case > > > > > > > > > (i.e. > > > > > > > > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going > to refrain from > > > > > > > > > > > releases > > > > > > > > > > > > > that support 1.20+ till the blockers are > implemented? > > > > > > > > > > > > > Best Regards > > > > > > > > > > > > > Ahmed Hamdy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 3 May 2024 at 14:32, Péter Váry < > > > > > peter.vary.apa...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> With regards to FLINK-35149, the fix version > indicates a change > > > > > at > > > > > > > > > > > Flink > > > > > > > > > > > > > >> CDC; is that indeed correct, or does it require > a change in the > > > > > > > SinkV2 > > > > > > > > > > > > > >> interface? > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> The fix doesn't need change in SinkV2, so we > are good there. > > > > > > > > > > > > > >> The issue is that the new SinkV2 > > > > > > > > > > > SupportsCommitter/SupportsPreWriteTopology > > > > > > > > > > > > > >> doesn't work with the CDC yet. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Martijn Visser ezt > írta (időpont: > > > > > 2024. > > > > > > > > > máj. > > > > > > > > > > > > > >> 3., > > > > > > > > > > > > > >> P, 14:06): > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >>> Hi Ferenc, > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> You're right, 1.20 it is :) > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> I've assigned the HBase one to you! > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Thanks, > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Martijn > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Thanks Ahmed! I have reviewed your PubSub and RabbitMQ PRs, please have a look. Best, Muhammet
Confirmation on Lambda Support for UDFs in FlinkSQL / Table API
Hi Flink Community, I’m writing to confirm whether lambda expressions are supported with User Defined Functions (UDFs) in FlinkSQL and the Table API. My current understanding is that they are not supported. Can anyone verify this, or let me know if there have been any recent changes regarding this? Thanks for your help. Best regards, Tucker
RE: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Xuannan, I like that you are cleaning up options that I assume are not recommended or currently used in some way. I have not got experience of these options. For the proposed deprecations, will there be a proposed recommended alternatives that will be mentioned in the deprecation. If they are going to be removed for v2, it would be good to explicitly document the thinking and impact of the deprecations and consequent removals, Kind regards, David. From: Xuannan Su Date: Tuesday, 14 May 2024 at 02:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0 Hi all, Thank you for all the comments and suggestions! If there are no further comments, I'd like to close the discussion and start the voting in two days. Best regards, Xuannan On Mon, May 13, 2024 at 3:10 PM Jeyhun Karimov wrote: > > Hi Xuannan, > > Thanks a lot for the update. The FLIP looks good to me. +1 for it. > > Regards, > Jeyhun > > On Mon, May 13, 2024 at 4:45 AM Xuannan Su wrote: > > > Hi Jeyhun, > > > > Thanks for the comment! > > > > Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated > > the FLIP to include the information. > > > > Best regards, > > Xuannan > > > > On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov > > wrote: > > > > > > Hi Xuannan, > > > > > > Thanks for driving this FLIP! > > > I have a minor comment. Do we plan to remove StreamPipelineOptions in > > 2.0, > > > as it only contains deprecated options? > > > > > > Regards, > > > Jeyhun > > > > > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > Thanks Xuannan for the update! > > > > > > > > LGTM, +1 for this proposal. > > > > > > > > Best, > > > > Rui > > > > > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su > > wrote: > > > > > > > > > Hi Rui, > > > > > > > > > > Thanks for the suggestion! > > > > > > > > > > I updated the description of > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > > > hard-coded it to 20. > > > > > > > > > > Best regards, > > > > > Xuannan > > > > > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> > > wrote: > > > > > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > > > > removed > > > > > > and hard-coded to either 10 or 20. > > > > > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > > > the overdraft buffer in the current FLIP? > > > > > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > > > - Removing this option means users cannot change it, it might be > > better > > > > > to > > > > > > turn it up. > > > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > > > doesn't > > > > > > introduce more risk. > > > > > > > > > > > > Best, > > > > > > Rui > > > > > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > > > wrote: > > > > > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > > > > > +1 for the proposal. > > > > > > > > > > > > > > Best, > > > > > > > Yuxin > > > > > > > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > > > solely > > > > > > > > changing the configuration options, but are bound to changing / > > > > > removal > > > > > > > of > > > > > > > > certain features. E.g., the removal of hash-blocking shuffle > > and > > > > > legacy > > > > > > > > hybrid shuffle mode, and the behavior change of overdraft > > network > > > > > > > buffers. > > > > > > > > Therefore, it might be nicer to provide an implementation plan > > > > with a > > > > > > > list > > > > > > > > of related tasks in the FLIP. This should not block the FLIP > > > > though. > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su < > > suxuanna...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, > > we > > > > have > > > > > > > > > revisited all runtime configurations and identified several > > > > > > > > > improvements to enhance user-friendliness and > > maintainability. In > > > > > this > > > > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > > > > you! > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Xuannan > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > >
[jira] [Created] (FLINK-35363) Reorganization of flink-connector-jdbc
João Boto created FLINK-35363: - Summary: Reorganization of flink-connector-jdbc Key: FLINK-35363 URL: https://issues.apache.org/jira/browse/FLINK-35363 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: João Boto Described in: [FLIP-449|https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35362) Reorganization of flink-connector-jdbc
João Boto created FLINK-35362: - Summary: Reorganization of flink-connector-jdbc Key: FLINK-35362 URL: https://issues.apache.org/jira/browse/FLINK-35362 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: João Boto Described in: [FLIP-449|https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35364) Create core module and move code
João Boto created FLINK-35364: - Summary: Create core module and move code Key: FLINK-35364 URL: https://issues.apache.org/jira/browse/FLINK-35364 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: João Boto * create core module * move all code to this new module as is * transforme flink-connector-jdbc in shaded module -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35365) Reorganize catalog and dialect code
João Boto created FLINK-35365: - Summary: Reorganize catalog and dialect code Key: FLINK-35365 URL: https://issues.apache.org/jira/browse/FLINK-35365 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: João Boto Reorganize code for catalog and dialect -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35366) Create all database modules
João Boto created FLINK-35366: - Summary: Create all database modules Key: FLINK-35366 URL: https://issues.apache.org/jira/browse/FLINK-35366 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: João Boto Create all database modules and move related code there -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35367) Reorganize datastream sink and source
João Boto created FLINK-35367: - Summary: Reorganize datastream sink and source Key: FLINK-35367 URL: https://issues.apache.org/jira/browse/FLINK-35367 Project: Flink Issue Type: Sub-task Reporter: João Boto Reorganize datastream sink and source -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35368) Reorganize table code
João Boto created FLINK-35368: - Summary: Reorganize table code Key: FLINK-35368 URL: https://issues.apache.org/jira/browse/FLINK-35368 Project: Flink Issue Type: Sub-task Reporter: João Boto -- This message was sent by Atlassian Jira (v8.20.10#820010)
[RESULT][VOTE] FLIP-449: Reorganization of flink-connector-jdbc
Hi, I am happy to say that FLIP-449: Reorganization of flink-connector-jdbc [1] has been accepted. The proposal has been accepted with 5 approving votes (1 binding) and there are no disapproval (voted on this thread [2]): Rui Fan (binding) Yuepeng Pan (non-binding) Aleksandr Pilipenko (non-binding) Muhammet Orazov (non-binding) Jeyhun Karimov (non-binding) [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc [2] https://lists.apache.org/thread/mfqx711zoxgs3sbojr2slqrt2xv2h5q9 Thanks to all involved. Best, Joao Boto
[jira] [Created] (FLINK-35369) Improve `Table API and SQL` page or add new page to guide new users to right Flink SQL option
Keith Lee created FLINK-35369: - Summary: Improve `Table API and SQL` page or add new page to guide new users to right Flink SQL option Key: FLINK-35369 URL: https://issues.apache.org/jira/browse/FLINK-35369 Project: Flink Issue Type: Improvement Components: Project Website Affects Versions: 1.19.0 Reporter: Keith Lee Attachments: LandscapeOfFlinkSQL.drawio(6).png Flink has rich and varied SQL offerings/deployment mode, it can take some time for new users to investigate and arrive at the right offering for them. Consider the available options: 1. Flink SQL Client (through SQL gateway, embedded or remote) 2. REST through SQL Gateway 3. A SQL client with Flink JDBC driver (through SQL gateway's REST interface) 4. A SQL client with Hive JDBC driver (through SQL gateway's HiveServer2 interface) 5. Flink Client submitting packaged application (Java/Scala/Python) 6. Submitting packaged archive with code that uses Table API (Additionally, Apache Zeppelin also provide notebook experience with its Flink SQL interpreter which builds upon Flink Client.) The improvement being suggested here is to either enrich existing [Table API and SQL overview page|https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/overview/] or create new page that contains the following information: 1. Diagram on the various options available (see diagram below) 2. Table explaining pros of each approach e.g. Flink SQL Client for initial experimentation, development, OLAP. Implementing on top of Flink SQL JDBC client or SQL Gateway REST for automation, HiveServer2 for inter-operabilty with Hive etc. The table will guide users to the corresponding page for each option. !LandscapeOfFlinkSQL.drawio(6).png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[DISCUSS] FLINK-35369: Improve `Table API and SQL` page or add new page to guide new users to right Flink SQL option
Hello everyone, I'd like to start a discussion on improving `Table API and SQL` overview page or add a new page to give new users quick overview of options available. I've opted for improvement Jira instead of a FLIP as the change does not affect public interfaces or Flink runtime. https://issues.apache.org/jira/browse/FLINK-35369 Appreciate your thoughts and suggestions. Best regards Keith Lee
[VOTE] Release flink-connector-opensearch v1.2.0, release candidate #1
Hi everyone, Please review and vote on release candidate #1 for flink-connector-opensearch v1.2.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], which are signed with the key with fingerprint F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag v1.2.0-rc1 [5], * website pull request listing the new release [6]. * CI build of the tag [7]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Note that this release is for Opensearch v1.x Thanks, Release Manager [1] https://issues.apache.org/jira/projects/FLINK/versions/12353812 [2] https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.2.0-rc1 [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1734 [5] https://github.com/apache/flink-connector-opensearch/releases/tag/v1.2.0-rc1 [6] https://github.com/apache/flink-web/pull/740 [7] https://github.com/apache/flink-connector-opensearch/actions/runs/9102334125
[VOTE] Release flink-connector-opensearch v2.0.0, release candidate #1
Hi everyone, Please review and vote on release candidate #1 for flink-connector-opensearch v2.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], which are signed with the key with fingerprint F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag v2.0.0-rc1 [5], * website pull request listing the new release [6]. * CI build of the tag [7]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Note that this release is for Opensearch v2.x Thanks, Release Manager [1] https://issues.apache.org/jira/projects/FLINK/versions/12354674 [2] https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-2.0.0-rc1 [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1735/ [5] https://github.com/apache/flink-connector-opensearch/releases/tag/v2.0.0-rc1 [6] https://github.com/apache/flink-web/pull/741 [7] https://github.com/apache/flink-connector-opensearch/actions/runs/9102980808
Re: [VOTE] Release flink-connector-opensearch v1.2.0, release candidate #1
+1 (non-binding), thanks Sergey! On Wed, May 15, 2024, 5:56 p.m. Sergey Nuyanzin wrote: > Hi everyone, > Please review and vote on release candidate #1 for > flink-connector-opensearch v1.2.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint > F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v1.2.0-rc1 [5], > * website pull request listing the new release [6]. > * CI build of the tag [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Note that this release is for Opensearch v1.x > > Thanks, > Release Manager > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353812 > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.2.0-rc1 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1734 > [5] > > https://github.com/apache/flink-connector-opensearch/releases/tag/v1.2.0-rc1 > [6] https://github.com/apache/flink-web/pull/740 > [7] > > https://github.com/apache/flink-connector-opensearch/actions/runs/9102334125 >
Re: [VOTE] Release flink-connector-opensearch v2.0.0, release candidate #1
+1 (non-binding), thanks Sergey! On Wed, May 15, 2024, 6:00 p.m. Sergey Nuyanzin wrote: > Hi everyone, > Please review and vote on release candidate #1 for > flink-connector-opensearch v2.0.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint > F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v2.0.0-rc1 [5], > * website pull request listing the new release [6]. > * CI build of the tag [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Note that this release is for Opensearch v2.x > > Thanks, > Release Manager > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12354674 > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-2.0.0-rc1 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1735/ > [5] > > https://github.com/apache/flink-connector-opensearch/releases/tag/v2.0.0-rc1 > [6] https://github.com/apache/flink-web/pull/741 > [7] > > https://github.com/apache/flink-connector-opensearch/actions/runs/9102980808 >
Re: [RESULT][VOTE] FLIP-449: Reorganization of flink-connector-jdbc
Thanks Joao Boto for driving the FLIP. We need 3 +1(binding) votes according to Flink Bylaws[1] before the community accepts it. You can search "Consensus" and "FLIP (Major Change)" keywords in this wiki. [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws Best, Rui On Thu, May 16, 2024 at 2:16 AM Joao Boto wrote: > Hi, > > I am happy to say that FLIP-449: Reorganization of flink-connector-jdbc [1] > has been accepted. > > The proposal has been accepted with 5 approving votes (1 binding) and there > are no disapproval (voted on this thread [2]): > Rui Fan (binding) > Yuepeng Pan (non-binding) > Aleksandr Pilipenko (non-binding) > Muhammet Orazov (non-binding) > Jeyhun Karimov (non-binding) > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc > [2] https://lists.apache.org/thread/mfqx711zoxgs3sbojr2slqrt2xv2h5q9 > > Thanks to all involved. > > Best, > Joao Boto >
Re: [VOTE] FLIP-450: Improve Runtime Configuration for Flink 2.0
+1 (binding) Best, Xintong On Wed, May 15, 2024 at 6:53 PM weijie guo wrote: > +1(binding) > > Best regards, > > Weijie > > > Rui Fan <1996fan...@gmail.com> 于2024年5月15日周三 17:50写道： > > > +1(binding) > > > > Best, > > Rui > > > > On Wed, May 15, 2024 at 5:01 PM Xuannan Su > wrote: > > > > > Hi everyone, > > > > > > Thanks for all the feedback about the FLIP-450: Improve Runtime > > > Configuration for Flink 2.0 [1] [2]. > > > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > > hours(excluding weekends,until MAY 20, 12:00AM GMT) unless there is an > > > objection or an insufficient number of votes. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > [2] https://lists.apache.org/thread/20mkzd31607posls793hxy7mht40xp2x > > > > > > > > > Best regards, > > > Xuannan > > > > > >
Re: [DISCUSSION] FLIP-457: Improve Table/SQL Configuration for Flink 2.0
Hi Jane, Thanks for driving this effort! And +1 for the proposed changes. I have one comment on the migration plan. For options to be moved to another module/package, I think we have to mark the old option deprecated in 1.20 for it to be removed in 2.0, according to the API compatibility guarantees[1]. We can introduce the new option in 1.20 with the same option key in the intended class. WDYT? Best, Xuannan [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees On Wed, May 15, 2024 at 6:20 PM Jane Chan wrote: > > Hi all, > > I'd like to start a discussion on FLIP-457: Improve Table/SQL Configuration > for Flink 2.0 [1]. This FLIP revisited all Table/SQL configurations to > improve user-friendliness and maintainability as Flink moves toward 2.0. > > I am looking forward to your feedback. > > Best regards, > Jane > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=307136992
Re:Confirmation on Lambda Support for UDFs in FlinkSQL / Table API
Hi, Tucker.
Could you provide some examples of how you use lambda expressions?
Currently, if you try to use a lambda expression as a member variable of a UDF
class, it will fail because UDF
classes are required to be serializable, and lambda expressions are not
serializable.
However, there are workarounds, such as marking the lambda member variable
function with the transient modifier
to prevent it from being serialized, and initializing the function in the
overridden open method.
I've provided an example below:
```
public class JavaFunc22 extends ScalarFunction {
transient Function f;
@Override
public void open(FunctionContext context) throws Exception {
f = str -> str + ":test";
}
public String eval(String s) {
return f.apply(s);
}
}
```
--
Best！
Xuyang
在 2024-05-15 23:49:50，"Tucker Harvey" 写道：
>Hi Flink Community,
>
>I’m writing to confirm whether lambda expressions are supported with User
>Defined Functions (UDFs) in FlinkSQL and the Table API. My current
>understanding is that they are not supported.
>Can anyone verify this, or let me know if there have been any recent changes
>regarding this?
>Thanks for your help.
>
>Best regards,
>Tucker
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi David, Thanks for the feedback! > For the proposed deprecations, will there be a proposed recommended > alternatives that will be mentioned in the deprecation. If they are going to > be removed for v2, it would be good to explicitly document the thinking and > impact of the deprecations and consequent removals, Some options to be deprecated are considered too complicated for user to tweak. There will be alternatives for those deprecated options, e.g., the option related to the floating buffer. While some options have a reasonable default value and there are hardly tweaked, those options will not have an alternative and the current default value will be used, e.g., the options related to the netty. And we will document the reason for the deprecation in 1.20. Best, Xuannan On Thu, May 16, 2024 at 12:01 AM David Radley wrote: > > Hi Xuannan, > I like that you are cleaning up options that I assume are not recommended or > currently used in some way. > > I have not got experience of these options. For the proposed deprecations, > will there be a proposed recommended alternatives that will be mentioned in > the deprecation. If they are going to be removed for v2, it would be good to > explicitly document the thinking and impact of the deprecations and > consequent removals, > Kind regards, David. > > From: Xuannan Su > Date: Tuesday, 14 May 2024 at 02:08 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration > for Flink 2.0 > Hi all, > > Thank you for all the comments and suggestions! If there are no > further comments, I'd like to close the discussion and start the > voting in two days. > > Best regards, > Xuannan > > > > On Mon, May 13, 2024 at 3:10 PM Jeyhun Karimov wrote: > > > > Hi Xuannan, > > > > Thanks a lot for the update. The FLIP looks good to me. +1 for it. > > > > Regards, > > Jeyhun > > > > On Mon, May 13, 2024 at 4:45 AM Xuannan Su wrote: > > > > > Hi Jeyhun, > > > > > > Thanks for the comment! > > > > > > Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated > > > the FLIP to include the information. > > > > > > Best regards, > > > Xuannan > > > > > > On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov > > > wrote: > > > > > > > > Hi Xuannan, > > > > > > > > Thanks for driving this FLIP! > > > > I have a minor comment. Do we plan to remove StreamPipelineOptions in > > > 2.0, > > > > as it only contains deprecated options? > > > > > > > > Regards, > > > > Jeyhun > > > > > > > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > > > Thanks Xuannan for the update! > > > > > > > > > > LGTM, +1 for this proposal. > > > > > > > > > > Best, > > > > > Rui > > > > > > > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su > > > wrote: > > > > > > > > > > > Hi Rui, > > > > > > > > > > > > Thanks for the suggestion! > > > > > > > > > > > > I updated the description of > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > > > > hard-coded it to 20. > > > > > > > > > > > > Best regards, > > > > > > Xuannan > > > > > > > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> > > > wrote: > > > > > > > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will > > > > > > > > be > > > > > > removed > > > > > > > and hard-coded to either 10 or 20. > > > > > > > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > > > > the overdraft buffer in the current FLIP? > > > > > > > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > > > > - Removing this option means users cannot change it, it might be > > > better > > > > > > to > > > > > > > turn it up. > > > > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > > > > doesn't > > > > > > > introduce more risk. > > > > > > > > > > > > > > Best, > > > > > > > Rui > > > > > > > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > > > > > > > +1 for the proposal. > > > > > > > > > > > > > > > > Best, > > > > > > > > Yuxin > > > > > > > > > > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > > > > solely > > > > > > > > > changing the configuration options, but are bound to changing > > > > > > > > > / > > > > > > removal > > > > > > > > of > > > > > > > > > certain features. E.g., the removal of hash-blocking shuffle > > > and > > > > > > legacy > > > > > > > > > hybrid shuffle mode, and the behavior change of overdraft > > > network > > > > > > > > buffers. > > > > > > > > > Therefore
[jira] [Created] (FLINK-35370) Create a temp module to test backward compatibility
João Boto created FLINK-35370: - Summary: Create a temp module to test backward compatibility Key: FLINK-35370 URL: https://issues.apache.org/jira/browse/FLINK-35370 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: João Boto -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
Ammar Master created FLINK-35371: Summary: Allow the keystore and truststore type to configured for SSL Key: FLINK-35371 URL: https://issues.apache.org/jira/browse/FLINK-35371 Project: Flink Issue Type: Improvement Components: Runtime / Network Reporter: Ammar Master Flink always creates a keystore and trustore using the [default type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236] defined in the JDK, which in most cases is JKS. We should add other configuration options to set the type explicitly to support other custom formats, and match the options provided by other applications by [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS] and [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format] already. The default would continue to be specified by the JDK. The SSLContext for the REST API can read the configuration option directly, and we need to add extra logic to the [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java] for Pekko. -- This message was sent by Atlassian Jira (v8.20.10#820010)
