[jira] [Created] (FLINK-32401) KafkaSourceBuilder reset the 'auto.offset.reset'

KianChen created FLINK-32401: Summary: KafkaSourceBuilder reset the 'auto.offset.reset' Key: FLINK-32401 URL: https://issues.apache.org/jira/browse/FLINK-32401 Project: Flink Issue Type: Improvem

Re: [DISCUSS] Flexible Custom Commit Policy with Parameter Passing

Hi, Yuxia. Thank you for your suggestion! I agree with your point and have made the necessary modification. If you have any additional feedback or further suggestions, please feel free to let me know. : ) Best regards, Bangui Dunn yuxia 于2023年6月21日周三 10:56写道： > Correct what I said in the previ

[DISCUSS] Release 2.0 Work Items

Hi devs, As previously discussed in [1], we had been collecting work item proposals for the 2.0 release until June 15th, on the wiki page [2]. - As we have passed the due date, I'd like to kindly remind everyone *not to add / remove items directly on the wiki page*. If needed, please post

Re: Async I/O: preserve stream order for requests on key level

+1 for this. Actually, this is a headache for Flink SQL too. There is certainly a lot of updated data (CDC changelog) in real stream processing, The semantics here is the need to ensure the order between keys, and different keys can be handled in disorder. I'm very happy that the community has a

Re: [DISCUSS] Flexible Custom Commit Policy with Parameter Passing

Correct what I said in the previous email: "But will it better to make it asList ? Something like:`.stringType().asList()`." => "But will it better to make it no default value? Something like:`.stringType().asList().noDefaultValue`." Best regards, Yuxia - 原始邮件 - 发件人: "yuxia" 收件人: "

Re: [DISCUSS] Flexible Custom Commit Policy with Parameter Passing

Hi, Bangui Dunn. Thanks for reaching us out. Generally + 1 for the configuration. But will it better to make it asList? Something like: `.stringType().asList()`. Best regards, Yuxia - 原始邮件 - 发件人: "Gary Ccc" 收件人: "dev" 发送时间: 星期二, 2023年 6 月 20日 下午 5:44:10 主题: [DISCUSS] Flexible Custom

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi, Shammon. Thanks for bring this FLIP. I share all the concerns with Qingsheng. Beside that, I have few question: 1: What kinds of JobStatus will the `JobExecutionStatusEven` including? From the jave doc "/** Job status changed event for runtime. */" for the `JobExecutionStatusEvent`, I beli

[jira] [Created] (FLINK-32400) KafkaSourceBuilder#maybeOverride get property as a string to checking if has value may not be the best way

KianChen created FLINK-32400: Summary: KafkaSourceBuilder#maybeOverride get property as a string to checking if has value may not be the best way Key: FLINK-32400 URL: https://issues.apache.org/jira/browse/FLINK-32400

[jira] [Created] (FLINK-32399) Fix flink-sql-connector-kinesis build for maven 3.8.6+

Aleksandr Pilipenko created FLINK-32399: --- Summary: Fix flink-sql-connector-kinesis build for maven 3.8.6+ Key: FLINK-32399 URL: https://issues.apache.org/jira/browse/FLINK-32399 Project: Flink

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Hi all, Thanks Xingtong for the summary. If I could only choose one of the given two options, I would go with option 1. I understood that option 2 worked great with Kafka. But the bridge release will still confuse users and my gut feeling is that many users will skip 2.0 and be waiting for 3.0 or

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi, Jing The default value for this ratio is a reference to other systems, such as Hive. As long as Runtime Filter can filter out more than half of the data, we can benefit from it. Of course, normally, as long as we can get the statistics, ndv are present, the use of rowCount should be less, so I

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Hi Xintong, Thanks for the summary, most of the points that you agreed upon also make sense to me. > 2. Dropping deprecated Public APIs in minor releases, or demoting APIs > from Public to PublicEvolving / Experimental / Retired in any version bump, > are not good practices. I hope we can

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Hi Xintong, Thanks for the update! FWIW, I'm also in agreement with the points that you and Becket agree on. As you can probably guess, I'd also vote for option (2) on how to handle removal of the DataStream API. This is actually similar to how Kafka handled the removal of Zookeeper, and it's

[jira] [Created] (FLINK-32398) Support Avro SpecificRecord in DataStream and Table conversion.

Jiangjie Qin created FLINK-32398: Summary: Support Avro SpecificRecord in DataStream and Table conversion. Key: FLINK-32398 URL: https://issues.apache.org/jira/browse/FLINK-32398 Project: Flink

Re: Async I/O: preserve stream order for requests on key level

Thank you very much! It seems like you have a quite similar goal. However, could you clarify: do you maintain the stream order on key level, or do you just limit the parallel requests per key to one without caring about the order? I'm not 100% sure how your implementation with futures is done. If

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

Following up, any interest in flink-metrics-prometheus? It is quite a stale package. I would be interested in contributing - time permitting. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify]

Re: [DISCUSS] Status of Statefun Project

Hi Galen, Yes, I'll be more than happy to help with Statefun releases. Best regards, Martijn On Tue, Jun 20, 2023 at 2:21 PM Galen Warren wrote: > Thanks. > > Martijn, to answer your question, I'd need to do a small amount of work to > get a PR ready, but not much. Happy to do it if we're dec

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

Becket and I had an offline voice call earlier today. We have reached consensus on some of the arguments, but still hold different opinions on some others. I'd like to post the outcome here for transparency. We both agree that: 1. Providing a migration period would be beneficial for our users,

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

Hi Chesnay, Thanks for your feedback. Comments inline Le 16/06/2023 à 17:24, Chesnay Schepler a écrit : 1) Options specific to the adaptive scheduler should start with "jobmanager.adaptive-scheduler". ok 2) There isn't /really /a notion of a "scaling event". The scheduler is informed abo

Re: Async I/O: preserve stream order for requests on key level

Hi Juho -- I'm doing something similar. In my case, I want to execute async requests concurrently for inputs associated with different keys but issue them sequentially for any given key. The way I do it is to create a keyed stream and use it as an input to an async function. In this arrangement, al

Re: [DISCUSS] Status of Statefun Project

Thanks. Martijn, to answer your question, I'd need to do a small amount of work to get a PR ready, but not much. Happy to do it if we're deciding to restart Statefun releases -- are we? -- Galen On Sat, Jun 17, 2023 at 9:47 AM Tzu-Li (Gordon) Tai wrote: > > Perhaps he could weigh in on whether

[jira] [Created] (FLINK-32397) Add doc for add/drop/show partition

luoyuxia created FLINK-32397: Summary: Add doc for add/drop/show partition Key: FLINK-32397 URL: https://issues.apache.org/jira/browse/FLINK-32397 Project: Flink Issue Type: Sub-task Co

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi Ron, Thanks for the clarification. That answered my questions. Regarding the ratio, since my gut feeling is that any value less than 0.8 or 0.9 won't help too much(I might be wrong). I was thinking of adapting the formula to somehow map the current 0.9-1 to 0-1, i.e. if user config 0.5, it wil

[jira] [Created] (FLINK-32396) Support timestamp for jdbc driver and gateway

Fang Yong created FLINK-32396: - Summary: Support timestamp for jdbc driver and gateway Key: FLINK-32396 URL: https://issues.apache.org/jira/browse/FLINK-32396 Project: Flink Issue Type: Improveme

Re: [DISCUSS] Graduate the FileSink to @PublicEvolving

Is this issue still unresolved? https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238 Based on prior discussion, I believe this could lead to data loss with FileSink. On Tue, Jun 20, 2023, 5:41 AM Jing Ge wrote: > Hi all, > > The FileSink has been marked as @Experimental[1]

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

Hi all, I've updated the FLIP according to the above discussion, the changes mainly include: Declare the supported shuffle types. Build the runtime filters using a two-phase approach. Add more potential future improvements. Best, Lijie liu ron 于2023年6月20日周二 11:19写道： > Hi, Jing > > Thanks for y

[jira] [Created] (FLINK-32395) EmbeddedFileSourceCsvReaderFormatTests.test_csv_add_columns_from fails on AZP

Sergey Nuyanzin created FLINK-32395: --- Summary: EmbeddedFileSourceCsvReaderFormatTests.test_csv_add_columns_from fails on AZP Key: FLINK-32395 URL: https://issues.apache.org/jira/browse/FLINK-32395 P

[jira] [Created] (FLINK-32394) Init.pos is partially stored in connector state

Usamah Jassat created FLINK-32394: - Summary: Init.pos is partially stored in connector state Key: FLINK-32394 URL: https://issues.apache.org/jira/browse/FLINK-32394 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi Shammon, Thanks for starting this FLIP! Data lineage is a very important topic, which has been missing for a long time in Flink. I have some questions about the FLIP. About events and listeners: 1. I’m not sure if it is necessary to expose JobType to in JobCreatedEvent. This is an internal cl

[jira] [Created] (FLINK-32393) NettyClientServerSslTest.testSslPinningForInvalidFingerprint fails with Address already in use

Sergey Nuyanzin created FLINK-32393: --- Summary: NettyClientServerSslTest.testSslPinningForInvalidFingerprint fails with Address already in use Key: FLINK-32393 URL: https://issues.apache.org/jira/browse/FLINK-32

[jira] [Created] (FLINK-32392) Several jobs failed on AZP with No space left on device

Sergey Nuyanzin created FLINK-32392: --- Summary: Several jobs failed on AZP with No space left on device Key: FLINK-32392 URL: https://issues.apache.org/jira/browse/FLINK-32392 Project: Flink

[jira] [Created] (FLINK-32391) YARNSessionFIFOITCase.checkForProhibitedLogContents fails on AZP

Sergey Nuyanzin created FLINK-32391: --- Summary: YARNSessionFIFOITCase.checkForProhibitedLogContents fails on AZP Key: FLINK-32391 URL: https://issues.apache.org/jira/browse/FLINK-32391 Project: Flink

[DISCUSS] Flexible Custom Commit Policy with Parameter Passing

To the Apache Flink Community Members, I hope this email finds you well. I am writing to discuss a potential improvement for the implementation of a custom commit policy in Flink. Background: We have encountered challenges in utilizing a custom commit policy due to the inability to pass parameters

[DISCUSS] Graduate the FileSink to @PublicEvolving

Hi all, The FileSink has been marked as @Experimental[1] since Oct. 2020. According to FLIP-197[2], I would like to propose to graduate it to @PublicEvloving in the upcoming 1.18 release. On the other hand, as a related topic, FileSource was marked as @PublicEvolving[3] 3 years ago. It deserves a

[jira] [Created] (FLINK-32390) [Dependabot] Bump socket.io-parser and socket.io and engine.io

Martijn Visser created FLINK-32390: -- Summary: [Dependabot] Bump socket.io-parser and socket.io and engine.io Key: FLINK-32390 URL: https://issues.apache.org/jira/browse/FLINK-32390 Project: Flink

[jira] [Created] (FLINK-32389) [Dependabot] Bump guava from 27.0.1-jre to 32.0.0-jre

Martijn Visser created FLINK-32389: -- Summary: [Dependabot] Bump guava from 27.0.1-jre to 32.0.0-jre Key: FLINK-32389 URL: https://issues.apache.org/jira/browse/FLINK-32389 Project: Flink Iss

[jira] [Created] (FLINK-32388) Add the ability to pass parameters to CUSTOM PartitionCommitPolicy

Gary Cao created FLINK-32388: Summary: Add the ability to pass parameters to CUSTOM PartitionCommitPolicy Key: FLINK-32388 URL: https://issues.apache.org/jira/browse/FLINK-32388 Project: Flink I

Re: [VOTE] FLIP-308: Support Time Travel

+1 (binding) Best Yun Tang From: John Roesler Sent: Tuesday, June 20, 2023 1:53 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-308: Support Time Travel Thanks for this FLIP, Feng! I've been following along, and I'm +1 (non-binding). Thanks, -John On 2023/0