[jira] [Created] (FLINK-33035) Add Transformer and Estimator for Als

weibo zhao created FLINK-33035: -- Summary: Add Transformer and Estimator for Als Key: FLINK-33035 URL: https://issues.apache.org/jira/browse/FLINK-33035 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-33036) Add Transformer and Estimator for Als

weibo zhao created FLINK-33036: -- Summary: Add Transformer and Estimator for Als Key: FLINK-33036 URL: https://issues.apache.org/jira/browse/FLINK-33036 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

Hi Jing, Thanks for the comments. 1. "For the batch cases, currently the BulkFormat for DataStream is > missing" - true, and there is another option to leverage > StreamFormatAdapter[1] > StreamFormatAdapter is internal and it requires a StreamFormat implementation for Avro files which does not e

[jira] [Created] (FLINK-33037) Bump Guava to 32.1.2-jre

Jing Ge created FLINK-33037: --- Summary: Bump Guava to 32.1.2-jre Key: FLINK-33037 URL: https://issues.apache.org/jira/browse/FLINK-33037 Project: Flink Issue Type: Improvement Reporter:

Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and kubernetes

After discussing this FLIP-334[1] offline with Gyula and Max, I updated the FLIP based on the latest conclusion. Big thanks to Gyula and Max for their professional advice! > Does the interface function of handlerRecommendedParallelism > in AutoScalerEventHandler conflict with > handlerScalingFail

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

Hi Tawfik, It's exciting to see any ongoing research that tries to push Flink forward! The get the discussion started, can you please your paper with the community? Assessing the proposal without further context is tough. Best, D. On Mon, Sep 4, 2023 at 4:42 PM Tawfek Yasser Tawfek wrote: > D

Re: Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

+1 since there is an alternative, more complete implementation available Best, D. On Sat, Sep 2, 2023 at 12:07 AM David Anderson wrote: > +1 > > Keeping the legacy implementation in place is confusing and encourages > adoption of something that really shouldn't be used. > > Thanks for driving t

[jira] [Created] (FLINK-33038) remove getMinRetentionTime in StreamExecDeduplicate

xiaogang zhou created FLINK-33038: - Summary: remove getMinRetentionTime in StreamExecDeduplicate Key: FLINK-33038 URL: https://issues.apache.org/jira/browse/FLINK-33038 Project: Flink Issue T

Re: [VOTE] Release flink-connector-hbase v3.0.0, release candidate 2

Hi, Thanks Martijn for initiating the release! +1 (non-binding) - checked signatures and checksums - checked source has no binaries - checked LICENSE and NOTICE files - approved web PR Cheers, Ferenc --- Original Message --- On Monday, September 4th, 2023 at 12:54, Samrat Deb wrot

[RESULT][VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

Hi everyone, The voting time for [VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable[1] has passed. I'm closing the vote now. There were 6 +1 votes, all were binding: - Martijn Visser (binding) - Benchao Li (binding) - Godfrey He (binding) - Sergey Nuyanzin (bind

[REQUEST] Edit Permissions for FLIP

Hi folks, I'm writing to request the edit permission for FLIP. My Confluence Wiki ID is: zhanghao.chen. I've recently reported two JIRA issues and was reminded of the need to create a FLIP for each of them as they would change the public API: 1. [FLINK-25371] Include data port as part of the

[DISSCUSS] Kubernetes Operator Flink Version Support Policy

Hi All! @Maximilian Michels has raised the question of Flink version support in the operator before the last release. I would like to open this discussion publicly so we can finalize this before the next release. Background: Currently the Flink Operator supports all Flink versions since Flink 1.

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

Sounds good to me, thanks. On Tue, Sep 5, 2023, 8:12 AM Gyula Fóra wrote: > Hi All! > > @Maximilian Michels has raised the question of Flink > version support in the operator before the last release. I would like to > open this discussion publicly so we can finalize this before the next > relea

[DISCUSS] FLIP-361: Improve GC Metrics

Hi Devs, I would like to start a discussion on FLIP-361: Improve GC Metrics [1]. The current Flink GC metrics [2] are not very useful for monitoring purposes as they require post processing logic that is also dependent on the current runtime environment. Problems: - Total time is not very relev

回复: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

+1 for the proposal. A side question: how will we handle a major Flink version given that Flink 2.0 is around the corner. Best, Zhanghao Chen 发件人: Gyula Fóra 发送时间: 2023年9月5日 20:12 收件人: dev 抄送: Maximilian Michels ; Thomas Weise ; Márton Balassi ; morh...@apache.

[jira] [Created] (FLINK-33039) Avro Specific Record Logical timestamp is not serialized in Parquet

Ahmed Elhassany created FLINK-33039: --- Summary: Avro Specific Record Logical timestamp is not serialized in Parquet Key: FLINK-33039 URL: https://issues.apache.org/jira/browse/FLINK-33039 Project: Fl

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

+1 (non-binding) Yuepeng Pan 于2023年9月1日周五 15:43写道： > +1 (non-binding) > > Best, > Yuepeng > > > > At 2023-09-01 14:32:19, "Jark Wu" wrote: > >+1 (binding) > > > >Best, > >Jark > > > >> 2023年8月30日 02:40，Venkatakrishnan Sowrirajan 写道： > >> > >> Hi everyone, > >> > >> Thank you all for your feedb

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

+1 (binding) On Tue, Sep 5, 2023 at 4:16 PM ConradJam wrote: > +1 (non-binding) > > Yuepeng Pan 于2023年9月1日周五 15:43写道： > > > +1 (non-binding) > > > > Best, > > Yuepeng > > > > > > > > At 2023-09-01 14:32:19, "Jark Wu" wrote: > > >+1 (binding) > > > > > >Best, > > >Jark > > > > > >> 2023年8月30日 0

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

+1 (non-binding) Best, Jiabao > 2023年9月5日 下午10:33，Martijn Visser 写道： > > +1 (binding) > > On Tue, Sep 5, 2023 at 4:16 PM ConradJam wrote: > >> +1 (non-binding) >> >> Yuepeng Pan 于2023年9月1日周五 15:43写道： >> >>> +1 (non-binding) >>> >>> Best, >>> Yuepeng >>> >>> >>> >>> At 2023-09-01 14:3

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

+1 (binding) On Tue, Sep 5, 2023 at 4:55 PM Jiabao Sun wrote: > +1 (non-binding) > > Best, > Jiabao > > > > 2023年9月5日 下午10:33，Martijn Visser 写道： > > > > +1 (binding) > > > > On Tue, Sep 5, 2023 at 4:16 PM ConradJam wrote: > > > >> +1 (non-binding) > >> > >> Yuepeng Pan 于2023年9月1日周五 15:43写道： >

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

+1, thanks for the proposal On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra wrote: > Hi All! > > @Maximilian Michels has raised the question of Flink > version support in the operator before the last release. I would like to > open this discussion publicly so we can finalize this before the next > re

Re: [DISCUSS] FLIP-361: Improve GC Metrics

Hi Gyula, +1 The proposed changes make sense and are in line with what is available for other metrics, e.g. number of records processed. -Max On Tue, Sep 5, 2023 at 2:43 PM Gyula Fóra wrote: > > Hi Devs, > > I would like to start a discussion on FLIP-361: Improve GC Metrics [1]. > > The current

Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and kubernetes

Thanks Rui for the update! Alongside with the refactoring to decouple autoscaler logic from the deployment logic, are we planning to add an alternative implementation against the new interfaces? I think the best way to get the interfaces right, is to have an alternative implementation in addition

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

+1 On Tue, Sep 5, 2023 at 8:03 AM Thomas Weise wrote: > +1, thanks for the proposal > > On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra wrote: > > > Hi All! > > > > @Maximilian Michels has raised the question of Flink > > version support in the operator before the last release. I would like to > > o

[jira] [Created] (FLINK-33040) flink-connector-hive builds might be blocked (but not fail) because Maven tries to access conjars.org repository (which times out)

Matthias Pohl created FLINK-33040: - Summary: flink-connector-hive builds might be blocked (but not fail) because Maven tries to access conjars.org repository (which times out) Key: FLINK-33040 URL: https://issues.

Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy

+1 Sounds good! Four releases give a decent amount of time to migrate to the next Flink version. On Tue, Sep 5, 2023 at 5:33 PM Őrhidi Mátyás wrote: > > +1 > > On Tue, Sep 5, 2023 at 8:03 AM Thomas Weise wrote: > > > +1, thanks for the proposal > > > > On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra

Re: [DISCUSS] FLIP-334 : Decoupling autoscaler and kubernetes

Hi Max, > are we planning to add an alternative implementation against the new interfaces? Yes, we are simultaneously working on the YARN implementation using the interface. During the initial interface design, we encountered some anomalies while implementing it in YARN. Once the interfaces are

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi Venkata, > Also I made minor changes to the *NestedFieldReferenceExpression, *instead > of *fieldIndexArray* we can just do away with *fieldNames *array that > includes fieldName at every level for the nested field. I don't think keeping only the field names array would work. At the end of t

Re: [DISCUSS] Drop python 3.7 support in 1.19

Hi Gabor, Thanks for bringing this up. In my opinion, it is a bit aggressive to directly drop Python 3.7 in 1.19. Python 3.7 is still used a lot[1], and as far as I know, many Pyflink users are still using python 3.7 as their default interpreter. I prefer to deprecate Python 3.7 in 1.19 just like

[jira] [Created] (FLINK-33041) Add an introduction about how to migrate DataSet API to DataStream

Wencong Liu created FLINK-33041: --- Summary: Add an introduction about how to migrate DataSet API to DataStream Key: FLINK-33041 URL: https://issues.apache.org/jira/browse/FLINK-33041 Project: Flink

Re: [DISCUSS] FLIP-361: Improve GC Metrics

Hi Gyula, +1 for this proposal. The current GC metric is really unfriendly. I have a concern with your proposed rate metric: the rate is perSecond instead of per minute. I'm unsure whether it's suitable for GC metric. There are two reasons why I suspect perSecond may not be well compatible with

Re: [DISCUSS] Add config to enable job stop with savepoint on exceeding tolerable checkpoint Failures

Hi Dongwoo, If the checkpoint has failed `execution.checkpointing.tolerable-failed-checkpoints` times, then stopWithSavepoint is likely to fail as well. If stopWithSavepoint succeeds or fails, will the job just stop? I am more curious about how this option works with the restart strategy? Best,

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

It looks like it will be nice to have a config to disable hints. Any other thoughts/concerns before we can close this discussion? On Fri, Aug 18, 2023 at 7:43 AM Timo Walther wrote: > > lots of the streaming SQL syntax are extensions of SQL standard > > That is true. But hints are kind of a spe

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

Hi, Tawfik Yasser. Thanks for the proposal. It sounds exciting. I can't wait the research paper for more details. Best regards, Yuxia - 原始邮件 - 发件人: "David Morávek" 收件人: "dev" 发送时间: 星期二, 2023年 9 月 05日 下午 4:36:51 主题: Re: Proposal for Implementing Keyed Watermarks in Apache Flink Hi Tawf

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Based on an offline discussion with Becket Qin, I added *fieldIndices *back which is the field index of the nested field at every level to the *NestedFieldReferenceExpression *in FLIP-356 *. *2 rea

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

Based on the recent discussions in the thread [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown , I made some changes to the FLIP-356

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

Thanks for pushing the FLIP through. +1 on the updated FLIP wiki. Cheers, Jiangjie (Becket) Qin On Wed, Sep 6, 2023 at 1:12 PM Venkatakrishnan Sowrirajan wrote: > Based on the recent discussions in the thread [DISCUSS] FLIP-356: Support > Nested Fields Filter Pushdown >

Re: [VOTE] FLIP-356: Support Nested Fields Filter Pushdown

+1 On Wed, Sep 6, 2023 at 1:18 PM Becket Qin wrote: > > Thanks for pushing the FLIP through. > > +1 on the updated FLIP wiki. > > Cheers, > > Jiangjie (Becket) Qin > > On Wed, Sep 6, 2023 at 1:12 PM Venkatakrishnan Sowrirajan > wrote: > > > Based on the recent discussions in the thread [DISCUSS]

Re: [DISCUSS] FLIP-361: Improve GC Metrics

Thanks for the feedback Rui, The rates would be computed using the MeterView class (like for any other rate metric), just because we report the value per second it doesn't mean that we measure in a second granularity. By default the meterview measures for 1 minute and then we calculate the per sec

Re: [DISCUSS] Drop python 3.7 support in 1.19

Hi Xingbo! I think we have to analyze what we gain by dropping 3.7 and upgrading to a miniconda version with a multiarch support. If this is what we need to get Apple silicon support then I think it's worth doing it already in 1.19. Keep in mind that 1.18 is not even released yet so if we delay t

Re: [DISCUSS] FLIP-361: Improve GC Metrics

Thanks for the clarification! By default the meterview measures for 1 minute sounds good to me! +1 for this proposal. Best, Rui On Wed, Sep 6, 2023 at 1:27 PM Gyula Fóra wrote: > Thanks for the feedback Rui, > > The rates would be computed using the MeterView class (like for any other > rate

Re: [DISCUSS] Drop python 3.7 support in 1.19

Hi Xingbo, *Constraint:* I personally not found any miniconda version which provides arm64 support together with python 3.7. [image: image.png] At the moment I think new platform support means 3.7 drop. I fully to agree with Gyula, if we start now maybe we can release it in half a year however *

Re: Proposal for Implementing Keyed Watermarks in Apache Flink

Hi Tawfik, Thanks for offering such a proposal, looking forward to your research paper! You could also ask the edit permission for Flink improvement proposals to create a new proposal if you want to contribute this to the community by yourself. [1] https://cwiki.apache.org/confluence/display/

Re: [DISCUSS] FLIP-361: Improve GC Metrics

Thanks for bringing this up, Gyula. The proposed changes make sense to me. +1 for them. In addition to the proposed changes, I wonder if we should also add something like timePerGc? This would help understand whether there are long pauses, due to GC STW, that may lead to rpc unresponsiveness and