Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-14 Thread Xintong Song
Hi Gyula, First of all, even if we remove the `getXXX(String key, XXX defaultValue)` methods, there are still several ways to access the configuration with string-keys. - If one wants to access a specific option, as Rui mentioned, `ConfigOptions.key("xxx").stringType().noDefaultValue()` can

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

2023-12-14 Thread Márton Balassi
+1 Thanks, Peter. Based on the consensus in the recent thread on FLIP-371 [1] I agree that this is the right approach. I made some minor edits to the FLIP, which looks good to me now. [1] https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57 On Wed, Dec 13, 2023 at 5:30 PM Gyula Fóra

Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-14 Thread Gyula Fóra
I see a strong value for user facing configs to use ConfigOption and this should definitely be an enforced convention. However with the Flink project growing and many other components and even users using the Configuration object I really don’t think that we should “force” this on the users/develo

[jira] [Created] (FLINK-33820) StreamingWithStateTestBase's subclasses might fail during cleanup if sharing the state directory (through a common @TempDir)

2023-12-14 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-33820: - Summary: StreamingWithStateTestBase's subclasses might fail during cleanup if sharing the state directory (through a common @TempDir) Key: FLINK-33820 URL: https://issues.apache

[DISCUSS] FLIP-387: Support named parameters for functions and call procedures

2023-12-14 Thread Feng Jin
Hi all, Xuyang and I would like to start a discussion of FLIP-387: Support named parameters for functions and call procedures [1] Currently, when users call a function or call a procedure, they must specify all fields in order. When there are a large number of parameters, it is easy to make mist

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-14 Thread Timo Walther
Hi Alan, thanks for proposing this FLIP. It's a great addition to Flink and has been requested multiple times. It will be in particular interesting for accessing REST endpoints and other remote services. Great that we can generalize and reuse parts of the Python planner rules and code for th

[jira] [Created] (FLINK-33821) ArrowSerializer$finishCurrentBatch consumes too much time

2023-12-14 Thread wuwenchi (Jira)
wuwenchi created FLINK-33821: Summary: ArrowSerializer$finishCurrentBatch consumes too much time Key: FLINK-33821 URL: https://issues.apache.org/jira/browse/FLINK-33821 Project: Flink Issue Type:

Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-14 Thread Timo Walther
The configuration in Flink is complicated and I fear we won't have enough capacity to substantially fix it. The introduction of ReadableConfig, WritableConfig, and typed ConfigOptions was a right step into making the code more maintainable. From the Flink side, every read access should go throu

Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-14 Thread Timo Walther
Hi Xuyang, > I'm not spliting this flip is that all of these subtasks like session window tvf and cdc support do not change the public interface and the public syntax Given the length of this mailing list discussion and number of involved people I would strongly suggest to simplify the FLIP

Re: [DISCUSS] FLIP-387: Support named parameters for functions and call procedures

2023-12-14 Thread Timo Walther
Hi Feng, thank you for proposing this FLIP. This nicely completes FLIP-65 which is great for usability. I read the FLIP and have some feedback: 1) ArgumentNames annotation > Deprecate the ArgumentNames annotation as it is not user-friendly for specifying argument names with optional config

Re: [DISCUSS] Release Flink 1.18.1

2023-12-14 Thread Jing Ge
Hi folks, What Martijn said makes sense. We should pay more attention to https://issues.apache.org/jira/browse/FLINK-33793. I have also tried to contribute and share my thoughts in the PR https://github.com/apache/flink/pull/23489. For 1.18.1-rc1, I will wait until tomorrow. If there is no progre

Re: Edit permissions request

2023-12-14 Thread Matthias Pohl
Sure, I will help you. On Wed, Dec 13, 2023 at 7:55 PM Alex Nitavsky wrote: > Hello, > > I would love to initiate FLIP in order to extend Apache Curator > configuration: https://issues.apache.org/jira/browse/FLINK-33376. > > Would it be please possible to get edit permissions for Apache Flink >

[jira] [Created] (FLINK-33822) Move Slack Invite URL into config.toml

2023-12-14 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-33822: - Summary: Move Slack Invite URL into config.toml Key: FLINK-33822 URL: https://issues.apache.org/jira/browse/FLINK-33822 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-33823) Serialize PlannerQueryOperation into SQL

2023-12-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-33823: Summary: Serialize PlannerQueryOperation into SQL Key: FLINK-33823 URL: https://issues.apache.org/jira/browse/FLINK-33823 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-387: Support named parameters for functions and call procedures

2023-12-14 Thread Feng Jin
Hi Timo Thanks for your reply. > 1) ArgumentNames annotation I'm sorry for my incorrect expression. argumentNames is a method of FunctionHints. We should introduce a new arguments method to replace this method and return Argument[]. I updated the FLIP doc about this part. > 2) Evolution of Fun

[jira] [Created] (FLINK-33826) CLONE - Triage release-blocking issues in JIRA

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33826: --- Summary: CLONE - Triage release-blocking issues in JIRA Key: FLINK-33826 URL: https://issues.apache.org/jira/browse/FLINK-33826 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33828) CLONE - Cross team testing

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33828: --- Summary: CLONE - Cross team testing Key: FLINK-33828 URL: https://issues.apache.org/jira/browse/FLINK-33828 Project: Flink Issue Type: Sub-task Reporter: J

[jira] [Created] (FLINK-33824) Prepare Flink 1.18.1 Release

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33824: --- Summary: Prepare Flink 1.18.1 Release Key: FLINK-33824 URL: https://issues.apache.org/jira/browse/FLINK-33824 Project: Flink Issue Type: New Feature Componen

[jira] [Created] (FLINK-33825) CLONE - Create a new version in JIRA

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33825: --- Summary: CLONE - Create a new version in JIRA Key: FLINK-33825 URL: https://issues.apache.org/jira/browse/FLINK-33825 Project: Flink Issue Type: Sub-task R

[jira] [Created] (FLINK-33831) CLONE - Create a release branch

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33831: --- Summary: CLONE - Create a release branch Key: FLINK-33831 URL: https://issues.apache.org/jira/browse/FLINK-33831 Project: Flink Issue Type: Sub-task Affects Versio

[jira] [Created] (FLINK-33832) CLONE - Verify that no exclusions were erroneously added to the japicmp plugin

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33832: --- Summary: CLONE - Verify that no exclusions were erroneously added to the japicmp plugin Key: FLINK-33832 URL: https://issues.apache.org/jira/browse/FLINK-33832 Project: Flink

[jira] [Created] (FLINK-33830) CLONE - Select executing Release Manager

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33830: --- Summary: CLONE - Select executing Release Manager Key: FLINK-33830 URL: https://issues.apache.org/jira/browse/FLINK-33830 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33827) CLONE - Review and update documentation

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33827: --- Summary: CLONE - Review and update documentation Key: FLINK-33827 URL: https://issues.apache.org/jira/browse/FLINK-33827 Project: Flink Issue Type: Sub-task Affect

[jira] [Created] (FLINK-33829) CLONE - Review Release Notes in JIRA

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33829: --- Summary: CLONE - Review Release Notes in JIRA Key: FLINK-33829 URL: https://issues.apache.org/jira/browse/FLINK-33829 Project: Flink Issue Type: Sub-task R

Re: join flink slack

2023-12-14 Thread Matthias Pohl
I updated the Slack invite link [1]. Feel free to use that one. I created a PR [2] to reflect the change on the website as well. Matthias [1] https://join.slack.com/t/apache-flink/shared_invite/zt-294plfx41-23lOoovZOdegKwjW9_0q_g [2] https://github.com/apache/flink-web/pull/704 On Wed, Dec 13, 2

[DISCUSS] FLIP-402: Extend ZooKeeper Curator configurations

2023-12-14 Thread Alex Nitavsky
Hi all, I would like to start a discussion thread for: *FLIP-402: Extend ZooKeeper Curator configurations *[1] * Problem statement * Currently Flink misses several Apache Curator configurations, which could be useful for Flink deployment with ZooKeeper as HA provider. * Proposed solution * We ha

[RESULT][VOTE] FLIP-401: REST API JSON response deserialization unknown field tolerance

2023-12-14 Thread Gabor Somogyi
Hi All, FLIP-401: REST API JSON response deserialization unknown field tolerance [1] has been accepted and voted through this thread [2]. The proposal received 4 binding approval votes and there is no disapproval: - Gyula Fora (binding) - Mátyás Őrhidi (binding) - Peter Huang (non-binding) - Maxi

Re: [VOTE] Release flink-connector-pulsar 4.1.0, release candidate #1

2023-12-14 Thread Hang Ruan
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with jdk8 - Verified web PR - Make sure flink-connector-base have the provided scope Best, Hang tison 于2023年12月14日周四 11:51写道: > Thanks Leonard for driving

[jira] [Created] (FLINK-33833) Build Release Candidate: 1.18.1-rc1

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33833: --- Summary: Build Release Candidate: 1.18.1-rc1 Key: FLINK-33833 URL: https://issues.apache.org/jira/browse/FLINK-33833 Project: Flink Issue Type: New Feature Affects

[jira] [Created] (FLINK-33835) CLONE - Stage source and binary releases on dist.apache.org

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33835: --- Summary: CLONE - Stage source and binary releases on dist.apache.org Key: FLINK-33835 URL: https://issues.apache.org/jira/browse/FLINK-33835 Project: Flink Issue Type

[jira] [Created] (FLINK-33834) CLONE - Build and stage Java and Python artifacts

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33834: --- Summary: CLONE - Build and stage Java and Python artifacts Key: FLINK-33834 URL: https://issues.apache.org/jira/browse/FLINK-33834 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33836) CLONE - Propose a pull request for website updates

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33836: --- Summary: CLONE - Propose a pull request for website updates Key: FLINK-33836 URL: https://issues.apache.org/jira/browse/FLINK-33836 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33837) CLONE - Vote on the release candidate

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33837: --- Summary: CLONE - Vote on the release candidate Key: FLINK-33837 URL: https://issues.apache.org/jira/browse/FLINK-33837 Project: Flink Issue Type: Sub-task Affects

[jira] [Created] (FLINK-33842) CLONE - Publish the Dockerfiles for the new release

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33842: --- Summary: CLONE - Publish the Dockerfiles for the new release Key: FLINK-33842 URL: https://issues.apache.org/jira/browse/FLINK-33842 Project: Flink Issue Type: Sub-tas

[jira] [Created] (FLINK-33838) Finalize release 1.18.1

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33838: --- Summary: Finalize release 1.18.1 Key: FLINK-33838 URL: https://issues.apache.org/jira/browse/FLINK-33838 Project: Flink Issue Type: New Feature Affects Versions: 1

[jira] [Created] (FLINK-33839) CLONE - Deploy Python artifacts to PyPI

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33839: --- Summary: CLONE - Deploy Python artifacts to PyPI Key: FLINK-33839 URL: https://issues.apache.org/jira/browse/FLINK-33839 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33841) CLONE - Create Git tag and mark version as released in Jira

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33841: --- Summary: CLONE - Create Git tag and mark version as released in Jira Key: FLINK-33841 URL: https://issues.apache.org/jira/browse/FLINK-33841 Project: Flink Issue Type

[jira] [Created] (FLINK-33840) CLONE - Deploy artifacts to Maven Central Repository

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33840: --- Summary: CLONE - Deploy artifacts to Maven Central Repository Key: FLINK-33840 URL: https://issues.apache.org/jira/browse/FLINK-33840 Project: Flink Issue Type: Sub-ta

[jira] [Created] (FLINK-33846) CLONE - Remove outdated versions

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33846: --- Summary: CLONE - Remove outdated versions Key: FLINK-33846 URL: https://issues.apache.org/jira/browse/FLINK-33846 Project: Flink Issue Type: Sub-task Repor

[jira] [Created] (FLINK-33843) Promote release 1.18.1

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33843: --- Summary: Promote release 1.18.1 Key: FLINK-33843 URL: https://issues.apache.org/jira/browse/FLINK-33843 Project: Flink Issue Type: New Feature Affects Versions: 1.

[jira] [Created] (FLINK-33849) CLONE - Update reference data for Migration Tests

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33849: --- Summary: CLONE - Update reference data for Migration Tests Key: FLINK-33849 URL: https://issues.apache.org/jira/browse/FLINK-33849 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33850) CLONE - Updates the docs stable version

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33850: --- Summary: CLONE - Updates the docs stable version Key: FLINK-33850 URL: https://issues.apache.org/jira/browse/FLINK-33850 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33847) CLONE - Apache mailing lists announcements

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33847: --- Summary: CLONE - Apache mailing lists announcements Key: FLINK-33847 URL: https://issues.apache.org/jira/browse/FLINK-33847 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33844) CLONE - Update japicmp configuration

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33844: --- Summary: CLONE - Update japicmp configuration Key: FLINK-33844 URL: https://issues.apache.org/jira/browse/FLINK-33844 Project: Flink Issue Type: Sub-task R

[jira] [Created] (FLINK-33845) CLONE - Merge website pull request

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33845: --- Summary: CLONE - Merge website pull request Key: FLINK-33845 URL: https://issues.apache.org/jira/browse/FLINK-33845 Project: Flink Issue Type: Sub-task Rep

[jira] [Created] (FLINK-33848) CLONE - Other announcements

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33848: --- Summary: CLONE - Other announcements Key: FLINK-33848 URL: https://issues.apache.org/jira/browse/FLINK-33848 Project: Flink Issue Type: Sub-task Reporter:

[jira] [Created] (FLINK-33851) CLONE - Start End of Life discussion thread for now outdated Flink minor version

2023-12-14 Thread Jing Ge (Jira)
Jing Ge created FLINK-33851: --- Summary: CLONE - Start End of Life discussion thread for now outdated Flink minor version Key: FLINK-33851 URL: https://issues.apache.org/jira/browse/FLINK-33851 Project: Flink

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

2023-12-14 Thread Ken Krugler
Hi Yong, Looks good, thanks for creating this. One comment - related to my recent email about Fury, I would love to see the v2 serialization decoupled from Kryo. As part of that, instead of using xxxKryo in methods, call them xxxGeneric. A more extreme change would be to totally rely on Fury (

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-14 Thread Alan Sheinberg
Thanks Piotr and Timo for your responses. To address your comments Timo: 1) Configuration Configuration keys like `table.exec.async-scalar.catalog.db.func-name.buffer-capacity` are currently not supported in the configuration stack. The key space > should remain constant. Only a constant key s

[VOTE] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

2023-12-14 Thread Wencong Liu
Hi dev, I'd like to start a vote on FLIP-382. Discussion thread: https://lists.apache.org/thread/3mgsc31odtpmzzl32s4oqbhlhxd0mn6b FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs Best regards, Wencong Liu

[VOTE] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-14 Thread Wencong Liu
Hi dev, I'd like to start a vote on FLIP-380. Discussion thread: https://lists.apache.org/thread/nn7myj7vsvytbkdrnbvj5h0homsjrn1h FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream Best regards, Wencong Liu

Re: [VOTE] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

2023-12-14 Thread weijie guo
+1(binding) Best regards, Weijie Wencong Liu 于2023年12月15日周五 09:13写道: > Hi dev, > > I'd like to start a vote on FLIP-382. > > Discussion thread: > https://lists.apache.org/thread/3mgsc31odtpmzzl32s4oqbhlhxd0mn6b > FLIP: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+

Re:Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-14 Thread Xuyang
Hi, Timo, thanks for your advice. I am considering splitting the existing flip into two while leaving the existing flip (or without). One of them points to the completion of the operator about window tvf to support CDC (there are several small work items, such as window agg, window rank, wind

Re: [VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-14 Thread Zhu Zhu
+1 (binding) Thanks, Zhu Xintong Song 于2023年12月14日周四 15:36写道: > +1 (binding) > > Best, > > Xintong > > > > On Thu, Dec 14, 2023 at 3:15 PM Lijie Wang > wrote: > > > Hi devs, Thanks for all feedback about the FLIP-383: Support Job Recovery > > for Batch Jobs[1]. This FLIP was discussed in [2].

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-14 Thread Aitozi
Hi Alan, Nice FLIP, I also explore leveraging the async table function[1] to improve the throughput before. About the configs, what do you think using hints as mentioned in [1]. [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction

Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-14 Thread Zhu Zhu
I think it's not clear whether forcing using ConfigOption would hurt the user experience. Maybe it does at the beginning, because using string keys to access Flink configuration can be simpler for new components/jobs. However, problems may happen later if the configuration usages become more compl

Re:Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-14 Thread Xuyang
Hi, Alan. Thanks for driving this. Using async to improve throughput has been done on look join, and the improvement effect is obvious, so I think it makes sense to support async scalar function. Big +1 for this flip. I have some questions below. 1. Override the function `getRequirements`

Re: [VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-14 Thread weijie guo
+1 (binding) Best regards, Weijie Zhu Zhu 于2023年12月15日周五 10:49写道: > +1 (binding) > > Thanks, > Zhu > > Xintong Song 于2023年12月14日周四 15:36写道: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Thu, Dec 14, 2023 at 3:15 PM Lijie Wang > > wrote: > > > > > Hi devs, Thanks for al

Re: [VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-14 Thread Rui Fan
+1 (binding) Best, Rui On Fri, Dec 15, 2023 at 12:05 PM weijie guo wrote: > +1 (binding) > > Best regards, > > Weijie > > > Zhu Zhu 于2023年12月15日周五 10:49写道: > > > +1 (binding) > > > > Thanks, > > Zhu > > > > Xintong Song 于2023年12月14日周四 15:36写道: > > > > > +1 (binding) > > > > > > Best, > > > >

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-14 Thread MengHui Yu
Good News. +1

[jira] [Created] (FLINK-33852) FLIP-403 High Availability Services for OLAP Scenarios

2023-12-14 Thread Yangze Guo (Jira)
Yangze Guo created FLINK-33852: -- Summary: FLIP-403 High Availability Services for OLAP Scenarios Key: FLINK-33852 URL: https://issues.apache.org/jira/browse/FLINK-33852 Project: Flink Issue Type

[jira] [Created] (FLINK-33853) Migrate Junit5 for DefaultDeclarativeSlotPool test classes

2023-12-14 Thread RocMarshal (Jira)
RocMarshal created FLINK-33853: -- Summary: Migrate Junit5 for DefaultDeclarativeSlotPool test classes Key: FLINK-33853 URL: https://issues.apache.org/jira/browse/FLINK-33853 Project: Flink Issue