Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-04-11 Thread Zhu Zhu
Hi Panagiotis, Thanks for updating the FLIP. > Regarding the config option `jobmanager.failure-enricher-plugins.enabled` I think a config option `jobmanager.failure-enrichers`, which accepts the names of enrichers to use, may be better. It allows the users to deploy and use the plugins in a more

[jira] [Created] (FLINK-31768) Create FLIP-285 interfaces without changing the behavior

2023-04-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31768: - Summary: Create FLIP-285 interfaces without changing the behavior Key: FLINK-31768 URL: https://issues.apache.org/jira/browse/FLINK-31768 Project: Flink Is

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-04-11 Thread Teoh, Hong
Hi Panagiotis, Thank you for the update. Looks great! Just one suggestion below: 1. We seem to be waiting for the future(s) to complete before restarting the job - should we add a configurable timeout for the enrichment? Since each failure enricher are run in parallel, we could probably settle

[jira] [Created] (FLINK-31769) Add percentiles to aggregated metrics

2023-04-11 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-31769: - Summary: Add percentiles to aggregated metrics Key: FLINK-31769 URL: https://issues.apache.org/jira/browse/FLINK-31769 Project: Flink Issue Type: Improveme

[jira] [Created] (FLINK-31770) OracleExactlyOnceSinkE2eTest.testInsert fails for JDBC connector

2023-04-11 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-31770: -- Summary: OracleExactlyOnceSinkE2eTest.testInsert fails for JDBC connector Key: FLINK-31770 URL: https://issues.apache.org/jira/browse/FLINK-31770 Project: Flink

Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement

2023-04-11 Thread yuxia
Hi, ron. 1: Considering for deleting rows, Flink will also write delete record to achive purpose of deleting data, it may not as so strange for connector devs to make DynamicTableSink implement SupportsTruncate to support truncate the table. Based on the assume that DynamicTableSink is used for

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-04-11 Thread Zakelly Lan
Hi Rui Fan, Thanks for your comments! > (1) The temporary segment will remain in the physical file for a short time, > right? Yes, any written segment will remain in the physical file until the physical file is deleted. It is controlled by the reference counting. And as discussed in 4.7, this w

[jira] [Created] (FLINK-31771) Improve select available slot from SlotPool

2023-04-11 Thread Weihua Hu (Jira)
Weihua Hu created FLINK-31771: - Summary: Improve select available slot from SlotPool Key: FLINK-31771 URL: https://issues.apache.org/jira/browse/FLINK-31771 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-31772) AsyncSinkWriter Performance regression due to AIMD rate limiting strategy

2023-04-11 Thread Ahmed Hamdy (Jira)
Ahmed Hamdy created FLINK-31772: --- Summary: AsyncSinkWriter Performance regression due to AIMD rate limiting strategy Key: FLINK-31772 URL: https://issues.apache.org/jira/browse/FLINK-31772 Project: Flin

[jira] [Created] (FLINK-31773) Introduces LeaderElection sub-interface

2023-04-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31773: - Summary: Introduces LeaderElection sub-interface Key: FLINK-31773 URL: https://issues.apache.org/jira/browse/FLINK-31773 Project: Flink Issue Type: Sub-tas

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-04-11 Thread Yun Tang
Hi Zakelly, Since we already had some discussions on this topic in the doc I mentioned, could you please describe the difference in your FLIP? I think we should better have a comparing table across different options just like the doc wrote. And we could also list some of them in your Rejected

[jira] [Created] (FLINK-31774) Add document for delete and update statement

2023-04-11 Thread Aitozi (Jira)
Aitozi created FLINK-31774: -- Summary: Add document for delete and update statement Key: FLINK-31774 URL: https://issues.apache.org/jira/browse/FLINK-31774 Project: Flink Issue Type: Improvement

Re: [VOTE] Release flink-connector-kafka, release candidate #1

2023-04-11 Thread Tzu-Li (Gordon) Tai
Hi all, Martijn and I discussed offline to cancel this vote. Moreover, now that Flink 1.17 is out and we still haven't released anything yet for the newly externalized Kafka connector, we've decided to skip releasing a version that matches with Flink 1.16 all together, and instead go straight to

Re: [VOTE] Release flink-connector-kafka, release candidate #1

2023-04-11 Thread Martijn Visser
+1, thanks for driving this Gordon. On Tue, Apr 11, 2023 at 8:15 PM Tzu-Li (Gordon) Tai wrote: > Hi all, > > Martijn and I discussed offline to cancel this vote. > > Moreover, now that Flink 1.17 is out and we still haven't released > anything yet for the newly externalized Kafka connector, we'v

[jira] [Created] (FLINK-31775) High-Availability not supported in kubernetes when istio enabled

2023-04-11 Thread Sergio Sainz (Jira)
Sergio Sainz created FLINK-31775: Summary: High-Availability not supported in kubernetes when istio enabled Key: FLINK-31775 URL: https://issues.apache.org/jira/browse/FLINK-31775 Project: Flink

Re: [DISCUSS] Add support for Apache Arrow format

2023-04-11 Thread Jacky Lau
Hi I also think arrow format will be useful when reading/writing with message queue. Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also sup

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-11 Thread John Roesler
Hi Jark, I hope you don’t mind if I chime in. You have a good point that the sequence of upserts will eventually converge to the correct value under the at-least-once delivery guarantee, but it can still be important to avoid passing on uncommitted results. Some thoughts, numbered for referenc

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-04-11 Thread Hongshun Wang
Hi everyone, I have already modified FLIP-288 to provide a newDiscoveryOffsetsInitializer in the KafkaSourceBuilder and KafkaSourceEnumerator. Users can use KafkaSourceBuilder#setNewDiscoveryOffsets to change the strategy for new partitions. Surely, enabling the partition discovery strategy by de