Re: Support local aggregate push down for Blink batch planner

Hi Sebastian, I assigned the issue to you. But I suggest creating sub-tasks under this issue. Because I think this would be a big contribution. For example, you can split it into: 1. Introduce SupportsAggregatePushDown interface 2. Support SupportsAggregatePushDown in planner 3. Support SupportsAg

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

Thanks for drafting the FLIP and driving the discussion, Yangze. And Thanks for the feedback, Till and Chesnay. @Till, I agree that specifying requirements for SSGs means that SSGs need to be supported in fine-grained resource management, otherwise each operator might use as many resources as the

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

Thanks for your feedback. @Till > the only option for a scheduler which does not support slot sharing groups is > to say that every operator in this slot sharing group needs a slot with the > same resources as the whole group. At the moment, all the implementations of the scheduler respect the s

[jira] [Created] (FLINK-20893) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel test failed with "TestTimedOutException"

Huang Xingbo created FLINK-20893: Summary: UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel test failed with "TestTimedOutException" Key: FLINK-20893 URL: https://issues.apache.or

Re: About JobMananger metrics scope

The only workaround is to add some constant to this option before deploying a Flink cluster. Relevant JIRA: https://issues.apache.org/jira/browse/FLINK-9543 On 1/7/2021 11:58 PM, Lu Niu wrote: Hi, Flink Dev First of all, Happy New Year! I have a question about JM monitoring. According to htt

Re: Re: Task scheduling of Flink

Hi Penguin, Jumping into this conversation since I worked on the same code base (Flink-1.11.2) for a recent project and might have a more fresh memory of the method calls. I believe the chain of methods you have highlighted is correct. Obviously, the chain would differ based on the configuration

About JobMananger metrics scope

Hi, Flink Dev First of all, Happy New Year! I have a question about JM monitoring. According to https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html, metrics.scope.jm only have one variable, which seems to be not enough for YARN deployment mode: 1. The metric doesn't contain

Kafka producer exactly once

Is there a Kafka producer that can do exactly once semantic without the use of transactions? Thanks

Re: Is development in FlinkML still active?

Thanks Badrul, Contribution to the ML pipeline is highly appreciated! Please don't hesitate to reach out if you got any questions. Cheers, Jiangjie (Becket) Qin On Fri, Jan 8, 2021 at 1:02 AM Badrul Chowdhury wrote: > Thanks all for the pointers! The architecture outlined in FLIP39 > < > http

Re: Is development in FlinkML still active?

Thanks all for the pointers! The architecture outlined in FLIP39 seems promising and the community seems to have accepted it- let me look into contributing to that effort. Thanks, Badrul On Thu, Jan 7, 2021

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Hi Roman, Very thanks for the feedbacks! I'll try to answer the issues inline: > 1. Option 1 is said to be not preferable because it wastes resources and adds > complexity (new event). > However, the resources would be wasted for a relatively short time until the > job finishes completely.

Re: [DISCUSS] FLIP-155: Introduce a few convenient operations in Table API

Hi Dian, Thanks for working on improving the Table API. I went through the entire FLIP and many functions definitely make sense. However, we need to make sure that the general API naming, behavior etc. remains consistent. Here is some feedback from my side: 1) deduplicate Are we planning to

Re: Support local aggregate push down for Blink batch planner

Hi Jark, Seems that we have reached the agreement on the proposal. Could you please help to assign the below jira ticket to me? https://issues.apache.org/jira/browse/FLINK-20791 Jark Wu 于2021年1月7日周四 上午10:25写道： > Thanks for updating the design doc. > It looks good to me. > > Best, > Jark > > On

[jira] [Created] (FLINK-20892) Remove ScheduledUnit interface

Matthias created FLINK-20892: Summary: Remove ScheduledUnit interface Key: FLINK-20892 URL: https://issues.apache.org/jira/browse/FLINK-20892 Project: Flink Issue Type: Sub-task Compone

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Thanks for starting this discussion (and sorry for probably duplicated questions, I couldn't find them answered in FLIP or this thread). 1. Option 1 is said to be not preferable because it wastes resources and adds complexity (new event). However, the resources would be wasted for a relatively sho

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

Will declaring them on slot sharing groups not also waste resources if the parallelism of operators within that group are different? It also seems like quite a hassle for users having to recalculate the resource requirements if they change the slot sharing. I'd think that it's not really workab

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

Thanks for drafting this FLIP and starting this discussion Yangze. I like that defining resource requirements on a slot sharing group makes the overall setup easier and improves usability of resource requirements. What I do not like about it is that it changes slot sharing groups from being a sch

[jira] [Created] (FLINK-20891) SystemResourcesCounterTest may caused endless loop when some error occured

zlzhang0122 created FLINK-20891: --- Summary: SystemResourcesCounterTest may caused endless loop when some error occured Key: FLINK-20891 URL: https://issues.apache.org/jira/browse/FLINK-20891 Project: Fl

[jira] [Created] (FLINK-20890) flink-state-processor-api: differents serializers being taken from Flink & State processor api

Felipe Silvestre Santos de Morais created FLINK-20890: - Summary: flink-state-processor-api: differents serializers being taken from Flink & State processor api Key: FLINK-20890 URL: https://iss

Re: Is development in FlinkML still active?

Hi Flavio, Currently most of the Flink AI efforts are in ecosystem projects. Primarily flink-ai-extended[1] and Alink[2] as you noticed. - flink-ai-extended aims to provide a solution to unify big data and AI (especially deep learning) based on Flink. - Alink is a classic machine learning library

[jira] [Created] (FLINK-20889) How does flink 1.12 support modifying jobName

HideOnBush created FLINK-20889: -- Summary: How does flink 1.12 support modifying jobName Key: FLINK-20889 URL: https://issues.apache.org/jira/browse/FLINK-20889 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

> > We could introduce an interface, sth like `RequiresFinalization` or > `FinalizationListener` (all bad names). The operator itself knows when > it is ready to completely shut down, Async I/O would wait for all > requests, sink would potentially wait for a given number of checkpoints. > The inter

[jira] [Created] (FLINK-20888) ContinuousFileReaderOperator should not close the output on close()

Yun Gao created FLINK-20888: --- Summary: ContinuousFileReaderOperator should not close the output on close() Key: FLINK-20888 URL: https://issues.apache.org/jira/browse/FLINK-20888 Project: Flink Is

[jira] [Created] (FLINK-20887) Non-deterministic functions return different values even if it is referred with the same column name

Caizhi Weng created FLINK-20887: --- Summary: Non-deterministic functions return different values even if it is referred with the same column name Key: FLINK-20887 URL: https://issues.apache.org/jira/browse/FLINK-20887

Re: Is development in FlinkML still active?

Or also https://github.com/alibaba/Alink, I don't know if the 2 are related somehow.. On Thu, Jan 7, 2021 at 9:55 AM Flavio Pompermaier wrote: > What about Flink-AI [1]? Would you suggest its adoption Till? > > [1] https://github.com/alibaba/flink-ai-extended > > On Thu, Jan 7, 2021 at 9:38 AM T

Re: [DISCUSS] Backport broadcast operations in BATCH mode to Flink

+1 on my side as it does not break anything and it can act as motivation for some people to upgrade. Cheers, Kostas On Thu, 7 Jan 2021, 12:39 Aljoscha Krettek, wrote: > 1.12.x > Reply-To: > > Hi, > > what do you think about backporting FLINK-20491 [1] to Flink 1.12.x? > > I (we, including Dawid

[DISCUSS] Backport broadcast operations in BATCH mode to Flink

1.12.x Reply-To: Hi, what do you think about backporting FLINK-20491 [1] to Flink 1.12.x? I (we, including Dawid and Kostas) are a bit torn on this. a) It's a limitation of Flink 1.12.0 and fixing this seems very good for users that would otherwise have to wait until Flink 1.13.0. b) It's t

[jira] [Created] (FLINK-20886) Add the option to get a threaddump on checkpoint timeouts

Nico Kruber created FLINK-20886: --- Summary: Add the option to get a threaddump on checkpoint timeouts Key: FLINK-20886 URL: https://issues.apache.org/jira/browse/FLINK-20886 Project: Flink Issue

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

This is somewhat unrelated to the discussion about how to actually do the triggering when sources shut down, I'll write on that separately. I just wanted to get this quick thought out. For letting operators decide whether they actually want to wait for a final checkpoint, which is relevant at

[jira] [Created] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column

Jark Wu created FLINK-20885: --- Summary: Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column Key: FLINK-20885 URL: https://issues.apache.org/jira/browse/FLINK-20885

[jira] [Created] (FLINK-20884) NullPointerException in create statements with computed columns which the subQuery is SqlNodeList

allan.hou created FLINK-20884: - Summary: NullPointerException in create statements with computed columns which the subQuery is SqlNodeList Key: FLINK-20884 URL: https://issues.apache.org/jira/browse/FLINK-20884

[jira] [Created] (FLINK-20883) Separate the implementation of BatchExecOverAggregate and StreamExecOverAggregate

godfrey he created FLINK-20883: -- Summary: Separate the implementation of BatchExecOverAggregate and StreamExecOverAggregate Key: FLINK-20883 URL: https://issues.apache.org/jira/browse/FLINK-20883 Project

[jira] [Created] (FLINK-20882) Add better error message for illegal import checks

Chesnay Schepler created FLINK-20882: Summary: Add better error message for illegal import checks Key: FLINK-20882 URL: https://issues.apache.org/jira/browse/FLINK-20882 Project: Flink Is

[jira] [Created] (FLINK-20881) flink sql and flink sink must be run in one stream

donglei created FLINK-20881: --- Summary: flink sql and flink sink must be run in one stream Key: FLINK-20881 URL: https://issues.apache.org/jira/browse/FLINK-20881 Project: Flink Issue Type: New Feat

[jira] [Created] (FLINK-20880) Turn Json String into Row

tonychan created FLINK-20880: Summary: Turn Json String into Row Key: FLINK-20880 URL: https://issues.apache.org/jira/browse/FLINK-20880 Project: Flink Issue Type: Improvement Componen

Re: Is development in FlinkML still active?

What about Flink-AI [1]? Would you suggest its adoption Till? [1] https://github.com/alibaba/flink-ai-extended On Thu, Jan 7, 2021 at 9:38 AM Till Rohrmann wrote: > HI Badrul, > > FlinkML is unfortunately no longer under active development. However, there > is some new effort to add a machine l

Re: Is development in FlinkML still active?

HI Badrul, FlinkML is unfortunately no longer under active development. However, there is some new effort to add a machine learning library to Flink [1]. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs Cheers, Till On Wed, Jan 6, 2021 at 7:11 PM Badru

Re: [VOTE] Release 1.12.1, release candidate #1

+1 for canceling the RC to fix FLINK-20632. Thanks a lot for driving this effort Xintong and Robert! Cheers, Till On Wed, Jan 6, 2021 at 1:01 PM Xintong Song wrote: > Thanks for driving the docker image efforts, Robert. > > +1 for canceling this RC. That should also give us the chance to fix >

[jira] [Created] (FLINK-20879) Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions

godfrey he created FLINK-20879: -- Summary: Use MemorySize type instead of String type for memory ConfigOption in ExecutionConfigOptions Key: FLINK-20879 URL: https://issues.apache.org/jira/browse/FLINK-20879

Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector

Seeing strong support from the Pulsar community and Arvid volunteering to mentor this effort resolved my concerns. I think the Pulsar connector will be a good addition to the project as it opens up more integrations. Cheers, Till On Wed, Jan 6, 2021 at 11:50 AM Arvid Heise wrote: > Hi Till, > >

[jira] [Created] (FLINK-20878) Flink 1.12作业job Name修改

HideOnBush created FLINK-20878: -- Summary: Flink 1.12作业job Name修改 Key: FLINK-20878 URL: https://issues.apache.org/jira/browse/FLINK-20878 Project: Flink Issue Type: Bug Components: Tabl