[jira] [Created] (FLINK-23216) RM keeps allocating and freeing slots after a TM lost until its heartbeat timeout

Gen Luo created FLINK-23216: --- Summary: RM keeps allocating and freeing slots after a TM lost until its heartbeat timeout Key: FLINK-23216 URL: https://issues.apache.org/jira/browse/FLINK-23216 Project: Flin

[RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Hi there, Since the voting time of FLIP-147[1] has passed, I'm closing the vote now. There were seven +1 votes ( 6 / 7 are bindings) and no -1 votes: - Dawid Wysakowicz (binding) - Piotr Nowojski(binding) - Jiangang Liu (binding) - Arvid Heise (binding) - Jing Zhang (binding) - Leonard Xu (non-b

[jira] [Created] (FLINK-23215) flink-table-code-splitter: NOTICE should in META-INF

Jingsong Lee created FLINK-23215: Summary: flink-table-code-splitter: NOTICE should in META-INF Key: FLINK-23215 URL: https://issues.apache.org/jira/browse/FLINK-23215 Project: Flink Issue Ty

[jira] [Created] (FLINK-23214) Make ShuffleMaster a cluster level shared service

Yingjie Cao created FLINK-23214: --- Summary: Make ShuffleMaster a cluster level shared service Key: FLINK-23214 URL: https://issues.apache.org/jira/browse/FLINK-23214 Project: Flink Issue Type: S

Re: [ANNOUNCE] Criteria for merging pull requests is updated

- SHOULD NOT use the GitHub UI to merge PRs Where was this discussed? On 7/2/2021 6:59 AM, Xintong Song wrote: Hi Flink committers, As previously discussed [1], the criteria for merging pull requests has been updated. A full version of guidelines can be found on the project wiki [2]. The fo

[ANNOUNCE] Criteria for merging pull requests is updated

Hi Flink committers, As previously discussed [1], the criteria for merging pull requests has been updated. A full version of guidelines can be found on the project wiki [2]. The following are some of the highlights. - MUST make sure passing the CI tests before merging PRs - SHOULD NOT use the Git

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Best, Guowei On Fri, Jul 2, 2021 at 11:56 AM Leonard Xu wrote: > +1, > > This may help https://issues.apache.org/jira/browse/FLINK-22764 < > https://issues.apache.org/jira/browse/FLINK-22764> > > Best, > Leonard > > > 在 2021年7月2日，10:17，JING ZHANG 写道： > > > > +1 (binding) > > > > >

[jira] [Created] (FLINK-23213) Remove ProcessFunctionOperation

Dian Fu created FLINK-23213: --- Summary: Remove ProcessFunctionOperation Key: FLINK-23213 URL: https://issues.apache.org/jira/browse/FLINK-23213 Project: Flink Issue Type: Sub-task Componen

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

Thanks all for the positive feedback. I have updated the wiki page [1], and will send an announcement in a separate thread, to draw more committers' attention. Moreover, I've opened FLINK-23212 where we can continue with the discussion around pure documentation changing PRs. Thank you~ Xintong

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1, This may help https://issues.apache.org/jira/browse/FLINK-22764 Best, Leonard > 在 2021年7月2日，10:17，JING ZHANG 写道： > > +1 (binding) > > > Arvid Heise 于2021年7月1日周四 下午3:10写道： > >> Looks good: +1 (binding) >> >> On Tue, Jun 29, 2021 at 5

[jira] [Created] (FLINK-23212) Skip code-wise tests for pure documentation changing PRs

Xintong Song created FLINK-23212: Summary: Skip code-wise tests for pure documentation changing PRs Key: FLINK-23212 URL: https://issues.apache.org/jira/browse/FLINK-23212 Project: Flink Issu

Re: [DISCUSS] Better user experience in the WindowAggregate upon Changelog (contains update message)

Thanks for the discussion, JING ZHANG. I like the first proposal since it is simple and consistent with dataStream API. It is helpful to add more docs about the special late case in WindowAggregate. Also, I expect the more flexible emit strategies later. Jark Wu 于2021年7月2日周五 上午10:33写道： > Sorry,

Re: [DISCUSS] Better user experience in the WindowAggregate upon Changelog (contains update message)

Sorry, I made a typo above. I mean I prefer proposal (1) that only needs to set `table.exec.emit.allow-lateness` to handle late events. `table.exec.emit.late-fire.delay` can be optional which is 0s by default. `table.exec.state.ttl` will not affect window state anymore, so window state is still cle

Re: [VOTE] FLIP-150: Introduce Hybrid Source

+1 (non-binding) On Thu, Jul 1, 2021 at 6:45 PM Elkhan Dadashov wrote: > +1 (non-binding) > > On 2021/07/01 05:49:44 蒋晓峰 wrote: > > Hi everyone, > > > > > > > > > > Thanks for all the feedback to Hybrid Source so far. Based on the > discussion[1] we seem to have consensus, so I would like to sta

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

+1 (binding) Arvid Heise 于2021年7月1日周四 下午3:10写道： > Looks good: +1 (binding) > > On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 wrote: > > > +1 (binding) > > > > Best > > liujiangang > > > > Piotr Nowojski 于2021年6月29日周二 上午2:05写道： > > > > > +1 (binding) > > > > > > Piotrek > > > > > > pon., 28 cze 2021 o 1

[jira] [Created] (FLINK-23211) An old interface method is used in this section of [Passing Options Factory to RocksDB].

Carl created FLINK-23211: Summary: An old interface method is used in this section of [Passing Options Factory to RocksDB]. Key: FLINK-23211 URL: https://issues.apache.org/jira/browse/FLINK-23211 Project: Fli

Re: [VOTE] FLIP-150: Introduce Hybrid Source

+1 (non-binding) On 2021/07/01 05:49:44 蒋晓峰 wrote: > Hi everyone, > > > > > Thanks for all the feedback to Hybrid Source so far. Based on the > discussion[1] we seem to have consensus, so I would like to start a vote on > FLIP-150 for which the FLIP has also been updated[2]. > > > > > Th

Re: Re: [VOTE] FLIP-150: Introduce Hybrid Source

+1 (non-binding) On 2021/07/01 14:32:58 Steven Wu wrote: > +1 (non-binding) > > On Thu, Jul 1, 2021 at 4:59 AM Thomas Weise wrote: > > > +1 (binding) > > > > > > On Thu, Jul 1, 2021 at 8:13 AM Arvid Heise wrote: > > > > > +1 (binding) > > > > > > Thank you and Thomas for driving this > > > >

Re: Job Recovery Time on TM Lost

Another side question, Shall we add metric to cover the complete restarting time (phase 1 + phase 2)? Current metric jm.restartingTime only covers phase 1. Thanks! Best Lu On Thu, Jul 1, 2021 at 12:09 PM Lu Niu wrote: > Thanks TIll and Yang for help! Also Thanks Till for a quick fix! > > I did

Re: Job Recovery Time on TM Lost

Thanks TIll and Yang for help! Also Thanks Till for a quick fix! I did another test yesterday. In this test, I intentionally throw exception from the source operator: ``` if (runtimeContext.getIndexOfThisSubtask() == 1 && errorFrenquecyInMin > 0 && System.currentTimeMillis() - last

Re: Job Recovery Time on TM Lost

Thanks TIll and Yang for help! Also Thanks Till for a quick fix! I did another test yesterday. In this test, I intentionally throw exception from the source operator: ``` if (runtimeContext.getIndexOfThisSubtask() == 1 && errorFrenquecyInMin > 0 && System.currentTimeMillis() - last

Re: [VOTE] FLIP-150: Introduce Hybrid Source

+1 (non-binding) On Thu, Jul 1, 2021 at 4:59 AM Thomas Weise wrote: > +1 (binding) > > > On Thu, Jul 1, 2021 at 8:13 AM Arvid Heise wrote: > > > +1 (binding) > > > > Thank you and Thomas for driving this > > > > On Thu, Jul 1, 2021 at 7:50 AM 蒋晓峰 wrote: > > > > > Hi everyone, > > > > > > > > >

[jira] [Created] (FLINK-23210) Running HA per-job cluster (hashmap, sync) end-to-end test failed on azure

Dawid Wysakowicz created FLINK-23210: Summary: Running HA per-job cluster (hashmap, sync) end-to-end test failed on azure Key: FLINK-23210 URL: https://issues.apache.org/jira/browse/FLINK-23210 Pr

Re: Job Recovery Time on TM Lost

A quick addition, I think with FLINK-23202 it should now also be possible to improve the heartbeat mechanism in the general case. We can leverage the unreachability exception thrown if a remote target is no longer reachable to mark an heartbeat target as no longer reachable [1]. This can then be co

[jira] [Created] (FLINK-23209) Timeout heartbeat if the heartbeat target is no longer reachable

Till Rohrmann created FLINK-23209: - Summary: Timeout heartbeat if the heartbeat target is no longer reachable Key: FLINK-23209 URL: https://issues.apache.org/jira/browse/FLINK-23209 Project: Flink

[jira] [Created] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

Jiayi Liao created FLINK-23208: -- Summary: Late processing timers need to wait 1ms at least to be fired Key: FLINK-23208 URL: https://issues.apache.org/jira/browse/FLINK-23208 Project: Flink Iss

Re: [DISCUSS] Better user experience in the WindowAggregate upon Changelog (contains update message)

Thanks Jing for bringing up this topic, The emit strategy configs are annotated as Experiential and not public on documentations. However, I see this is a very useful feature which many users are looking for. I have posted these configs for many questions like "how to handle late events in SQL". T

[jira] [Created] (FLINK-23207) Flink Jira Bot to Ignore Tickets with fixVersion set

Konstantin Knauf created FLINK-23207: Summary: Flink Jira Bot to Ignore Tickets with fixVersion set Key: FLINK-23207 URL: https://issues.apache.org/jira/browse/FLINK-23207 Project: Flink

[jira] [Created] (FLINK-23206) Flink Jira Bot Moves Tickets to "Not a Priority" Instead of Closing Them

Konstantin Knauf created FLINK-23206: Summary: Flink Jira Bot Moves Tickets to "Not a Priority" Instead of Closing Them Key: FLINK-23206 URL: https://issues.apache.org/jira/browse/FLINK-23206 Proj

[jira] [Created] (FLINK-23205) Relax Time Intervals of Flink Jira Bot

Konstantin Knauf created FLINK-23205: Summary: Relax Time Intervals of Flink Jira Bot Key: FLINK-23205 URL: https://issues.apache.org/jira/browse/FLINK-23205 Project: Flink Issue Type: Im

Re: [VOTE] FLIP-150: Introduce Hybrid Source

+1 (binding) On Thu, Jul 1, 2021 at 8:13 AM Arvid Heise wrote: > +1 (binding) > > Thank you and Thomas for driving this > > On Thu, Jul 1, 2021 at 7:50 AM 蒋晓峰 wrote: > > > Hi everyone, > > > > > > > > > > Thanks for all the feedback to Hybrid Source so far. Based on the > > discussion[1] we se

Re: Job Recovery Time on TM Lost

Since you are deploying Flink workloads on Yarn, the Flink ResourceManager should get the container completion event after the heartbeat of Yarn NM->Yarn RM->Flink RM, which is 8 seconds by default. And Flink ResourceManager will release the dead TaskManager container once received the completion e

[DISCUSS] FLIP-173: Support DAG of algorithms (Flink ML)

Hi all, Zhipeng, Fan (cc'ed) and I are opening this thread to discuss two different designs to extend Flink ML API to support more use-cases, e.g. expressing a DAG of preprocessing and training logics. These two designs have been documented in FLIP-173

[VOTE] Release 1.13.2, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 1.13.2, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: [DISCUSS] Feedback Collection Jira Bot

It is true that the bot surfaces problems that are there (not enough committer attention sometimes), but it also "rubs salt in the wound" of contributors, and that is tricky. We can try it out with the extended periods (although I think that in reality we probably need even longer periods) and see

[jira] [Created] (FLINK-23204) Provide StateBackends access to MailboxExecutor

Roman Khachatryan created FLINK-23204: - Summary: Provide StateBackends access to MailboxExecutor Key: FLINK-23204 URL: https://issues.apache.org/jira/browse/FLINK-23204 Project: Flink Iss

[jira] [Created] (FLINK-23203) Program cannot parse the parameter value with special characters

Jacob.Q.Cao created FLINK-23203: --- Summary: Program cannot parse the parameter value with special characters Key: FLINK-23203 URL: https://issues.apache.org/jira/browse/FLINK-23203 Project: Flink

Improvements of left state in TemporalRowTimeJoinOperator

Hi Experts, Recently, I was learning how temporal table join works in Flink via reading the source code of TemporalRowTimeJoinOperator. and I found these comments in the source code: /** > * Mapping from artificial row index (generated by `nextLeftIndex`) > into the left side `Row`. We >

Re: Job Recovery Time on TM Lost

The analysis of Gen is correct. Flink currently uses its heartbeat as the primary means to detect dead TaskManagers. This means that Flink will take at least `heartbeat.timeout` time before the system recovers. Even if the cancellation happens fast (e.g. by having configured a low akka.ask.timeout)

[jira] [Created] (FLINK-23202) RpcService should fail result futures if messages could not be sent

Till Rohrmann created FLINK-23202: - Summary: RpcService should fail result futures if messages could not be sent Key: FLINK-23202 URL: https://issues.apache.org/jira/browse/FLINK-23202 Project: Flink

[jira] [Created] (FLINK-23201) The check on alignmentDurationNanos seems to be too strict

Jun Qin created FLINK-23201: --- Summary: The check on alignmentDurationNanos seems to be too strict Key: FLINK-23201 URL: https://issues.apache.org/jira/browse/FLINK-23201 Project: Flink Issue Type:

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

Looks good: +1 (binding) On Tue, Jun 29, 2021 at 5:06 AM 刘建刚 wrote: > +1 (binding) > > Best > liujiangang > > Piotr Nowojski 于2021年6月29日周二 上午2:05写道： > > > +1 (binding) > > > > Piotrek > > > > pon., 28 cze 2021 o 12:48 Dawid Wysakowicz > > napisał(a): > > > > > +1 (binding) > > > > > > Best, >