Re: Re: Re: [DISCUSS ] Make state.backend.incremental as true by default

Hi Yuan, I would like to share two cents here. No matter what results we can achieve, we would all agree that the configuration at the specific state-backend level should be prioritized than the overall `state.backend.incremental` config. With the above rules, let's take a close look at the imp

Re: Usage of PublicEvolving annotation

Hi all, let me add a few more words on this topic: The @PublicEvolving interface is kind of a staging annotation for @Public. We still need to be careful when to change classes with @PublicEvolving annotation. Usually, it still involves a proper deprecation process over 1-2 releases to give d

[jira] [Created] (FLINK-28504) Local-Global aggregation causes watermark alignment (table.exec.source.idle-timeout) of idle partition invalid

nyingping created FLINK-28504: - Summary: Local-Global aggregation causes watermark alignment (table.exec.source.idle-timeout) of idle partition invalid Key: FLINK-28504 URL: https://issues.apache.org/jira/browse/FLINK

[ANNOUNCE] Flink 1.16 feature freeze extended until August 9th, 2022

Hi everyone, As was already brought up for discussion [1] the release managers have decided in today's release sync meeting that we'll extend the Flink 1.16 feature freeze until August 9th, 2022. That gives us 4 weeks until the feature freeze begins. There will be no extension after August 9th, 2

[ACTION REQUESTED] Help needed with test stabilities

Hi everyone, Also discussed in today's release sync meeting are the number of test stabilities that are currently open and don't have yet a volunteer that works on them. You can find the list of the critical ones on the Flink release sync agenda [1]. We would like to request all contributors to s

[jira] [Created] (FLINK-28505) Pulsar sink doesn't support topic auto creation

Yufan Sheng created FLINK-28505: --- Summary: Pulsar sink doesn't support topic auto creation Key: FLINK-28505 URL: https://issues.apache.org/jira/browse/FLINK-28505 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-28506) Pulsar sink don't support zero timestamp which may be provided by flink

Yufan Sheng created FLINK-28506: --- Summary: Pulsar sink don't support zero timestamp which may be provided by flink Key: FLINK-28506 URL: https://issues.apache.org/jira/browse/FLINK-28506 Project: Flink

[jira] [Created] (FLINK-28507) supports set hadoop mount path in pod

jackylau created FLINK-28507: Summary: supports set hadoop mount path in pod Key: FLINK-28507 URL: https://issues.apache.org/jira/browse/FLINK-28507 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal

Thanks zhaobo for the proposal. We've met with a lot of big data users who are trying to migrate their business to flink on kubernetes to improve job performance and cluster utilization. Unfortunately so far kubernetes lacks batch scheduling capabilities compared to traditional batch systems like

[jira] [Created] (FLINK-28508) Support SPLIT_INDEX and STR_TO_MAP built-in function in Table API

LuNing Wang created FLINK-28508: --- Summary: Support SPLIT_INDEX and STR_TO_MAP built-in function in Table API Key: FLINK-28508 URL: https://issues.apache.org/jira/browse/FLINK-28508 Project: Flink

[jira] [Created] (FLINK-28509) Support REVERSE built-in function in Table API

LuNing Wang created FLINK-28509: --- Summary: Support REVERSE built-in function in Table API Key: FLINK-28509 URL: https://issues.apache.org/jira/browse/FLINK-28509 Project: Flink Issue Type: Impr

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Matyas, Thanks for the feedback, and yes I agree. An alternative approach would instead be: - 2 API calls only when jobID is not available (i.e when submitting a new application cluster, which is a one-off event). - 1 API call when jobID is already available by directly calling "/jobs/:jobid

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Devs! I discussed with Daren offline, and I agree with him that technically we almost never need 2 API calls. I think it's fine to have a second API call once directly after application submission (technically even this can be eliminated by setting a fix job id always). +1 from me. Cheers, G

Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution

+1 (binding) Thanks, Zhu Gen Luo 于2022年7月12日周二 13:46写道： > > Hi everyone, > > > Thanks for all the feedback so far. Based on the discussion [1], we seem to > have consensus. So, I would like to start a vote on FLIP-249 [2]. > > > The vote will last for at least 72 hours unless there is an objecti

[jira] [Created] (FLINK-28510) Support using new KafkaSink in PyFlink

Juntao Hu created FLINK-28510: - Summary: Support using new KafkaSink in PyFlink Key: FLINK-28510 URL: https://issues.apache.org/jira/browse/FLINK-28510 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-28511) Improve Spark dependencies and document

Jingsong Lee created FLINK-28511: Summary: Improve Spark dependencies and document Key: FLINK-28511 URL: https://issues.apache.org/jira/browse/FLINK-28511 Project: Flink Issue Type: Improveme

[jira] [Created] (FLINK-28512) Select HashBasedDataBuffer and SortBasedDataBuffer dynamically based on the number of network buffers can be allocated for

Yingjie Cao created FLINK-28512: --- Summary: Select HashBasedDataBuffer and SortBasedDataBuffer dynamically based on the number of network buffers can be allocated for Key: FLINK-28512 URL: https://issues.apache.org/j

[jira] [Created] (FLINK-28513) Flink Table API CSV streaming sink throws SerializedThrowable exception

Jaya Ananthram created FLINK-28513: -- Summary: Flink Table API CSV streaming sink throws SerializedThrowable exception Key: FLINK-28513 URL: https://issues.apache.org/jira/browse/FLINK-28513 Project:

[jira] [Created] (FLINK-28514) Remove data flush in SortMergeResultPartition

Yingjie Cao created FLINK-28514: --- Summary: Remove data flush in SortMergeResultPartition Key: FLINK-28514 URL: https://issues.apache.org/jira/browse/FLINK-28514 Project: Flink Issue Type: Sub-t

[DISCUSS] Kubernetes Operator 1.1.0 release

Hi Devs! Based on the previously agreed release schedule for the operator the next target release date is Aug 1. According to the schedule July 11 would have been Feature Freeze and Jul 18 the time of release cut.

[jira] [Created] (FLINK-28515) The files in local recovery directory hasn't be clean up properly after checkpoint abort

Jinzhong Li created FLINK-28515: --- Summary: The files in local recovery directory hasn't be clean up properly after checkpoint abort Key: FLINK-28515 URL: https://issues.apache.org/jira/browse/FLINK-28515

Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution

+1 (non-binding) Best, Lijie Zhu Zhu 于2022年7月12日周二 17:38写道： > +1 (binding) > > Thanks, > Zhu > > Gen Luo 于2022年7月12日周二 13:46写道： > > > > Hi everyone, > > > > > > Thanks for all the feedback so far. Based on the discussion [1], we seem > to > > have consensus. So, I would like to start a vote on

Re: [DISCUSS] Kubernetes Operator 1.1.0 release

Thanks Gyula, I'm happy to help with anything if needed. Best, Matyas On Tue, Jul 12, 2022 at 1:25 PM Gyula Fóra wrote: > Hi Devs! > > Based on the previously agreed release schedule for the operator > < > https://cwiki.apache.org/confluence/display/FLINK/Release+Schedule+and+Planning > > > th

[jira] [Created] (FLINK-28516) UI is misleading when source doesn't emit watermark due to misconfigured idleness

Chesnay Schepler created FLINK-28516: Summary: UI is misleading when source doesn't emit watermark due to misconfigured idleness Key: FLINK-28516 URL: https://issues.apache.org/jira/browse/FLINK-28516

[jira] [Created] (FLINK-28517) Bump Flink version to 1.15.1

Matyas Orhidi created FLINK-28517: - Summary: Bump Flink version to 1.15.1 Key: FLINK-28517 URL: https://issues.apache.org/jira/browse/FLINK-28517 Project: Flink Issue Type: Improvement Af

[jira] [Created] (FLINK-28518) Exception: AssertionError: Cannot add expression of different type to set in sub-query with ROW type

Koylubaev Nikita created FLINK-28518: Summary: Exception: AssertionError: Cannot add expression of different type to set in sub-query with ROW type Key: FLINK-28518 URL: https://issues.apache.org/jira/browse/F

[ANNOUNCE] Apache Flink ML 2.1.0 released

The Apache Flink community is excited to announce the release of Flink ML 2.1.0! This release focuses on improving Flink ML's infrastructure, such as Python SDK, memory management, and benchmark framework, to facilitate the development of performant, memory-safe, and easy-to-use algorithm librari

[jira] [Created] (FLINK-28519) Fix the bug that SortMergeResultPartitionReadScheduler may not read data sequentially

Yingjie Cao created FLINK-28519: --- Summary: Fix the bug that SortMergeResultPartitionReadScheduler may not read data sequentially Key: FLINK-28519 URL: https://issues.apache.org/jira/browse/FLINK-28519 P

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Thanks for the explanation. Only having 1 API call in most cases makes sense to me. Could you please elaborate more about why do we need the *plan* in CR status? Best, Yang Gyula Fóra 于2022年7月12日周二 17:36写道： > Hi Devs! > > I discussed with Daren offline, and I agree with him that technically w

[jira] [Created] (FLINK-28520) RestClient doesn't use SNI TLS extension

Jean-Damien HATZENBUHLER created FLINK-28520: Summary: RestClient doesn't use SNI TLS extension Key: FLINK-28520 URL: https://issues.apache.org/jira/browse/FLINK-28520 Project: Flink

Re: [DISCUSS] FLIP-249: Flink Web UI Enhancement for Speculative Execution

Thanks for driving this discussion. It's a very helpful improvement. I only have two minor questions: 1. Does the improvement also cover history server or just Web UI? 2. How to know whether the job contains speculative execution instances after the job finished? Do we have to check each subtasks o

Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution

Hi, Gen Luo, I left two minor questions in the DISCUSS thread. Sorry for jumping into the discussion so late. Best, Jing Zhang Lijie Wang 于2022年7月12日周二 19:29写道： > +1 (non-binding) > > Best, > Lijie > > Zhu Zhu 于2022年7月12日周二 17:38写道： > > > +1 (binding) > > > > Thanks, > > Zhu > > > > Gen Luo

[jira] [Created] (FLINK-28521) Hostname verification in ssl context is not working

Jean-Damien HATZENBUHLER created FLINK-28521: Summary: Hostname verification in ssl context is not working Key: FLINK-28521 URL: https://issues.apache.org/jira/browse/FLINK-28521 Project: F

Re: [DISCUSS] Add new JobStatus fields to Flink Kubernetes Operator CRD

Hi Yang, The requirement to add *plan* currently originates from an internal AWS requirement as our service needs visibility of *plan*, but we think it could be beneficial as well to customers who uses *plan* too. Regards, Daren On 12/07/2022, 13:23, "Yang Wang" wrote: CAUTION: This

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Hi Sebastian, Thanks for picking this up. > 5. There is NO configuration option to enable watermark alignment of splits or disable pause/resume capabilities. Isn't this contradicting what we actually agreed on? > we are planning to have a configuration based way to revert to the previous behavi

[jira] [Created] (FLINK-28522) [JUnit5 Migration] Module: flink-sequence-file

Ryan Skraba created FLINK-28522: --- Summary: [JUnit5 Migration] Module: flink-sequence-file Key: FLINK-28522 URL: https://issues.apache.org/jira/browse/FLINK-28522 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-28523) LocalRecoveryITCase failed with AssertionError

Huang Xingbo created FLINK-28523: Summary: LocalRecoveryITCase failed with AssertionError Key: FLINK-28523 URL: https://issues.apache.org/jira/browse/FLINK-28523 Project: Flink Issue Type: Bu

[jira] [Created] (FLINK-28524) Translate "User-defined Sources & Sinks"

hunter created FLINK-28524: -- Summary: Translate "User-defined Sources & Sinks" Key: FLINK-28524 URL: https://issues.apache.org/jira/browse/FLINK-28524 Project: Flink Issue Type: Improvement

RE: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal

+1 On 2022/07/07 01:15:13 bo zhaobo wrote: > Hi, all. > > I would like to raise a discussion in Flink dev ML about Support Customized > Kubernetes Schedulers. > Currentlly, Kubernetes becomes more and more polular for Flink Cluster > deployment, and its ability is better, especially, it supports

[jira] [Created] (FLINK-28525) HiveDialectITCase.testTableWithSubDirsInPartitionDir failed with AssertJMultipleFailuresError

Huang Xingbo created FLINK-28525: Summary: HiveDialectITCase.testTableWithSubDirsInPartitionDir failed with AssertJMultipleFailuresError Key: FLINK-28525 URL: https://issues.apache.org/jira/browse/FLINK-28525

[jira] [Created] (FLINK-28526) Fail to lateral join with UDTF from Table with timstamp column

Xuannan Su created FLINK-28526: -- Summary: Fail to lateral join with UDTF from Table with timstamp column Key: FLINK-28526 URL: https://issues.apache.org/jira/browse/FLINK-28526 Project: Flink I

[jira] [Created] (FLINK-28527) Fail to lateral join with UDTF from Table with timstamp column

Xuannan Su created FLINK-28527: -- Summary: Fail to lateral join with UDTF from Table with timstamp column Key: FLINK-28527 URL: https://issues.apache.org/jira/browse/FLINK-28527 Project: Flink I

[jira] [Created] (FLINK-28528) Table.getSchema fails on table with watermark

Xuannan Su created FLINK-28528: -- Summary: Table.getSchema fails on table with watermark Key: FLINK-28528 URL: https://issues.apache.org/jira/browse/FLINK-28528 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-28529) ChangelogPeriodicMaterializationSwitchStateBackendITCase.testSwitchFromDisablingToEnablingInClaimMode failed with CheckpointException: Checkpoint expired before completi

Huang Xingbo created FLINK-28529: Summary: ChangelogPeriodicMaterializationSwitchStateBackendITCase.testSwitchFromDisablingToEnablingInClaimMode failed with CheckpointException: Checkpoint expired before completing Key: FLINK-285

[jira] [Created] (FLINK-28530) Improvement of conditions extraction that can be pushed into join inputs

Alexander Trushev created FLINK-28530: - Summary: Improvement of conditions extraction that can be pushed into join inputs Key: FLINK-28530 URL: https://issues.apache.org/jira/browse/FLINK-28530 Pr

[jira] [Created] (FLINK-28531) Shutdown cluster after history server archive finished

Aitozi created FLINK-28531: -- Summary: Shutdown cluster after history server archive finished Key: FLINK-28531 URL: https://issues.apache.org/jira/browse/FLINK-28531 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-28532) Support InputFormat as full caching provider in lookup table source

Qingsheng Ren created FLINK-28532: - Summary: Support InputFormat as full caching provider in lookup table source Key: FLINK-28532 URL: https://issues.apache.org/jira/browse/FLINK-28532 Project: Flink

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Hi Piotrek, Sorry I've read it and forgot it when I was ripping out the supportsPauseOrResume method again. Thanks for pointing that out. I will add it as follows: The enables/disables split alignment in the SourceOperator where the default is that split alignment is enabled. (And I will add the

Re: [DISCUSS] FLIP-249: Flink Web UI Enhancement for Speculative Execution

Hi Jing, Thanks for joining the discussion. It's a very good point to figure out the possible influence on the history server. > 1. Does the improvement also cover history server or just Web UI? As far as I know most Web UI components are shared between runtime and history server, so the improvem

Re: [VOTE] FLIP-249: Flink Web UI Enhancement for Speculative Execution

Hi Jing, I have replied in the discussion thread about the questions. Hope that would be helpful. Best, Gen On Tue, Jul 12, 2022 at 8:43 PM Jing Zhang wrote: > Hi, Gen Luo, > > I left two minor questions in the DISCUSS thread. > Sorry for jumping into the discussion so late. > > Best, > Jing