[jira] [Created] (FLINK-23377) Flink on Yarn could not set priority

2021-07-13 Thread HideOnBush (Jira)
HideOnBush created FLINK-23377: -- Summary: Flink on Yarn could not set priority Key: FLINK-23377 URL: https://issues.apache.org/jira/browse/FLINK-23377 Project: Flink Issue Type: Bug Co

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-13 Thread Steven Wu
I am trying to understand what those two metrics really capture > G setPendingBytesGauge(G pendingBytesGauge); - use file source as an example, it captures the remaining bytes for the current file split that the reader is processing? How would users interpret or use this metric? enumera

Re: [DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-13 Thread Steven Wu
- The subtask observes the changes in the throughput and changes the buffer size during the whole life period of the task. - The subtask sends buffer size and number of available buffers to the upstream to the corresponding subpartition. - Upstream changes the buffer size correspondi

[jira] [Created] (FLINK-23376) Flink history server not showing running applications

2021-07-13 Thread kevinsun (Jira)
kevinsun created FLINK-23376: Summary: Flink history server not showing running applications Key: FLINK-23376 URL: https://issues.apache.org/jira/browse/FLINK-23376 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-23375) Flink connector jdbc tests jar is almost empty

2021-07-13 Thread Jira
Maciej Bryński created FLINK-23375: -- Summary: Flink connector jdbc tests jar is almost empty Key: FLINK-23375 URL: https://issues.apache.org/jira/browse/FLINK-23375 Project: Flink Issue Type

Re: [DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-13 Thread Yingjie Cao
Hi, Thanks for driving this, I think it is really helpful for jobs suffering from backpressure. Best, Yingjie Anton,Kalashnikov 于2021年7月9日周五 下午10:59写道: > Hey! > > There is a wish to decrease amount of in-flight data which can improve > aligned checkpoint time(fewer in-flight data to process be

[jira] [Created] (FLINK-23374) Clean up logs produced by code splitter

2021-07-13 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-23374: --- Summary: Clean up logs produced by code splitter Key: FLINK-23374 URL: https://issues.apache.org/jira/browse/FLINK-23374 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-13 Thread Arvid Heise
Hi Becket, I believe 1+2 has been answered by Chesnay already. Just to add to 2: I'm not the biggest fan of reusing task metrics but that's what FLIP-33 and different folks suggested. I'd probably keep task I/O metrics only for internal things and add a new metric for external calls. Then, we coul

Re: [DISCUSS] FLIP-182: Watermark alignment

2021-07-13 Thread Stephan Ewen
@Eron Wright The per-split watermarks are the default in the new source interface (FLIP-27) and come for free if you use the SplitReader. Based on that, it is also possible to unsubscribe individual splits to solve the alignment in the case where operators have multiple splits assigned. Piotr an

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

2021-07-13 Thread Senhong Liu
Hi Stephan, Thank you so much for all these practical suggestions. I agree that hybrid batch/streaming execution is an awesome idea and that no checkpoint could be successful during the batch phase. However, the checkpoint could fail frequently because of the large batch size, which might be unne

[NOTICE] flink-runtime now scala-free

2021-07-13 Thread Chesnay Schepler
Hello everyone, I just merged the last PR for FLINK-14105, with which flink-runtime is now officially scala-free. *fireworks* What does that mean in practice? a) flink-runtime no longer has a scala-suffix, which cascaded into other modules (e.g., our reporter modules). This _may_ cau

[DISCUSS] Address deprecation warnings when upgrading dependencies

2021-07-13 Thread Stephan Ewen
Hi all! I would like to propose that we make it a project standard that when upgrading a dependency, deprecation issues arising from that need to be fixed in the same step. If the new dependency version deprecates a method in favor of another method, all usages in the code need to be replaced toge

[jira] [Created] (FLINK-23373) Support object reuse disabled in OperatorChain

2021-07-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-23373: Summary: Support object reuse disabled in OperatorChain Key: FLINK-23373 URL: https://issues.apache.org/jira/browse/FLINK-23373 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-23372) Disable AllVerticesInSameSlotSharingGroupByDefault in DataStream batch mode

2021-07-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-23372: Summary: Disable AllVerticesInSameSlotSharingGroupByDefault in DataStream batch mode Key: FLINK-23372 URL: https://issues.apache.org/jira/browse/FLINK-23372 Project:

Re: [DISCUSS] Releasing Flink 1.11.4

2021-07-13 Thread Till Rohrmann
Hi Godfrey, Are you continuing with the 1.11.4 release process? Cheers, Till On Tue, Jul 6, 2021 at 1:15 PM Chesnay Schepler wrote: > Since 1.11.4 is about releasing the commits we already have merged > between 1.11.3 and 1.13.0, I would suggest to not add additional fixes. > > On 06/07/2021 1

Re: [VOTE] Release 1.13.2, release candidate #1

2021-07-13 Thread Till Rohrmann
Hi everyone, FLINK-23233 has been merged. We can continue with the release process. Cheers, Till On Wed, Jul 7, 2021 at 2:00 PM Yun Tang wrote: > Since FLINK-23233 brings concerns to many guys, and this release candidate > #1 had not received enough binding +1 from PMCs, I think we could cance

Re: [VOTE] Release 1.12.5, release candidate #1

2021-07-13 Thread Till Rohrmann
Hi everyone, FLINK-23233 has been merged. We can continue with the release process. Cheers, Till On Wed, Jul 7, 2021 at 1:29 PM Jingsong Li wrote: > Hi all, > > Thanks Xiongtong, Leonard, Yang, JING for the voting. Thanks Till for the > information. > > +1 for canceling this RC. That should al

[jira] [Created] (FLINK-23371) Disable AutoWatermarkInterval for bounded legacy sources

2021-07-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-23371: Summary: Disable AutoWatermarkInterval for bounded legacy sources Key: FLINK-23371 URL: https://issues.apache.org/jira/browse/FLINK-23371 Project: Flink Issu

[jira] [Created] (FLINK-23370) Propagate Boundedness of SourceFunctionProvider to Transformation

2021-07-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-23370: Summary: Propagate Boundedness of SourceFunctionProvider to Transformation Key: FLINK-23370 URL: https://issues.apache.org/jira/browse/FLINK-23370 Project: Flink

Re: [DISCUSS]FLIP-170 Adding Checkpoint Rejection Mechanism

2021-07-13 Thread Stephan Ewen
Before jumping into the designs of other mechanisms, can we clarify whether adjusting checkpointing is really the right approach here? What about storing empty state in the checkpoints, so that any recovery simply does a full replay of the input. Regarding the hybrid batch/streaming execution: Th

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-13 Thread Chesnay Schepler
Re 1: We don't expose the reuse* methods, because the proposed OperatorIOMetricGroup is a separate interface from the existing implementations (which will be renamed and implement the new interface). Re 2: Currently the plan is to re-use the "new" numByesIn/Out counters for tasks ("new" becaus

[jira] [Created] (FLINK-23369) Use enums for connector options

2021-07-13 Thread Jira
Ingo Bürk created FLINK-23369: - Summary: Use enums for connector options Key: FLINK-23369 URL: https://issues.apache.org/jira/browse/FLINK-23369 Project: Flink Issue Type: Sub-task Re

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-13 Thread Becket Qin
Hi Arvid, Thanks for the proposal. I like the idea of exposing concrete metric group class so that users can access the predefined metrics. A few questions are following: 1. When exposing the OperatorIOMetrics to the users, we are also exposing the reuseInputMetricsForTask to the users. Should w

Re: [ANNOUNCE] New Apache Flink Committer - Yang Wang

2021-07-13 Thread Yang Wang
Thanks to all of you. Best, Yang XING JIN 于2021年7月8日周四 上午11:46写道: > Congratulations Yang Wang! > > Austin Cawley-Edwards 于2021年7月7日周三 下午9:29写道: > > > Congrats Yang! > > > > On Wed, Jul 7, 2021 at 7:26 AM Jingsong Li > wrote: > > > > > Congratulations, Yang! > > > > > > Best, > > > Jingsong >

[DISCUSS] FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework

2021-07-13 Thread Yingjie Cao
Hi devs and users, This topic was originally discussed and reached a consensus in [1]. Because the change touches the pluggable shuffle interface, though not annotated as public currently, some users may be using it already. To avoid bring compatibility issues to customized shuffle plugins already

[jira] [Created] (FLINK-23368) Fix the wrong mapping of state cache in PyFlink

2021-07-13 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-23368: Summary: Fix the wrong mapping of state cache in PyFlink Key: FLINK-23368 URL: https://issues.apache.org/jira/browse/FLINK-23368 Project: Flink Issue Type: B