[jira] [Created] (FLINK-14381) Partition field names should be got from CatalogTable instead of source/sink

2019-10-11 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14381: Summary: Partition field names should be got from CatalogTable instead of source/sink Key: FLINK-14381 URL: https://issues.apache.org/jira/browse/FLINK-14381 Project:

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

2019-10-11 Thread zhijiang
Sorry for delay catching up with the recent progress. Thanks for the FLIP update and valuable discussions! I also like the term of job/cluster partitions, and agree with most of the previous comments. Only left one concern of ShuffleMaster side: >However, if the separation of JM/RM into separat

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-11 Thread Jingsong Li
Hi Stephan, big +1 for this contribution. It provides another user interface that is easy to use and popular at this time. these functions, It's hard for users to write in SQL/TableApi, while using DataStream is too complex. (We've done some stateFun kind jobs using DataStream before). With statef

Re: [DISCUSS] Flink Python UDF Environment and Dependency Management

2019-10-11 Thread Dian Fu
Hi Wei, Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE. Thanks, Dian > 在 2019年10月9日,上午11:23,Wei Zhong

[jira] [Created] (FLINK-14380) Type Extractor POJO setter check does not allow for Immutable Case Class

2019-10-11 Thread Elliot Pourmand (Jira)
Elliot Pourmand created FLINK-14380: --- Summary: Type Extractor POJO setter check does not allow for Immutable Case Class Key: FLINK-14380 URL: https://issues.apache.org/jira/browse/FLINK-14380 Projec

[jira] [Created] (FLINK-14379) Supplement documentation about raw state

2019-10-11 Thread Yun Tang (Jira)
Yun Tang created FLINK-14379: Summary: Supplement documentation about raw state Key: FLINK-14379 URL: https://issues.apache.org/jira/browse/FLINK-14379 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14378) Cleanup rocksDB lib folder if fail to load library

2019-10-11 Thread Yun Tang (Jira)
Yun Tang created FLINK-14378: Summary: Cleanup rocksDB lib folder if fail to load library Key: FLINK-14378 URL: https://issues.apache.org/jira/browse/FLINK-14378 Project: Flink Issue Type: Improv

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-11 Thread Zili Chen
Well. Then I'd remove the requirement to change cancelWithSavepoint but remain why we exclude it from JobClient. We might still change signature to completable future for a consistent async view of ClusterClient but it is quite an implement detail and we don't stick to it on FLIP level. Best, tis

[jira] [Created] (FLINK-14377) Translate ProgramOptions relevant for job execution to ConfigOptions.

2019-10-11 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-14377: -- Summary: Translate ProgramOptions relevant for job execution to ConfigOptions. Key: FLINK-14377 URL: https://issues.apache.org/jira/browse/FLINK-14377 Project: Fl

[jira] [Created] (FLINK-14376) Introduce the Executor abstraction (FLIP-73)

2019-10-11 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-14376: -- Summary: Introduce the Executor abstraction (FLIP-73) Key: FLINK-14376 URL: https://issues.apache.org/jira/browse/FLINK-14376 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

2019-10-11 Thread Dawid Wysakowicz
Thank you all for the discussion. I added a note that we might revisit the signature with an instance. I started a separate voting thread for the current state of the FLIP. I will appreciate your votes. Best, Dawid On 11/10/2019 13:21, Jark Wu wrote: > Hi Timo, > > Sounds good to me. > > Thanks

[VOTE] FLIP-64: Support for Temporary Objects in Table module

2019-10-11 Thread Dawid Wysakowicz
Hi everyone, I would like to start a vote on FLIP-64. The discussion seems to have reached an agreement. Please vote for the following design document: https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module The discussion can be found at: htt

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-11 Thread Kostas Kloudas
Hi Tison, Thanks for integrating the comments! +1 for accepting the FLIP from my side. What I meant is that in the Proposed Changes section, the FLIP still has that the cancelWithSavepoin(jobId, savepointDir) of the clusterClient should change to return a CompletableFuture. I believe that this ch

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-11 Thread Jark Wu
Hi Stephan, big +1 for adding statefun to Apache Flink. I think we can add it in the main repository as a library just like State Process API. Having it in the main repository can closely co-develop with Apache Flink which be beneficial for both side. Regards, Jark On Fri, 11 Oct 2019 at 19:12

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

2019-10-11 Thread Jark Wu
Hi Timo, Sounds good to me. Thanks, Jark On Fri, 11 Oct 2019 at 17:37, Timo Walther wrote: > Hi Jark, > > great to see that there is consensus on something like `CREATE FUNCTION > bf1 WITH (precision=0.001)`. > > In the near future, there should be no difference between a catalog > function an

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-11 Thread Till Rohrmann
Hi Stephan, +1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike. One reason for not including it in the main repository would to not being bound to Flink's release cadence. This

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-11 Thread Trevor Grant
+1 non-binding on contribution. Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02. On Fri, Oct 11, 2019 at 4:38 AM Stephan

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-11 Thread Piotr Nowojski
Hi Stephan, +1 for adding this to Apache Flink! Regarding the question if this should be committed to the main repository or as a separate one, I think it should be the main one. Previously we were discussing the idea of splitting Apache Flink into multiple repositories and I think the consen

[jira] [Created] (FLINK-14375) Avoid to trigger failover on a non-effective task failure notification

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14375: --- Summary: Avoid to trigger failover on a non-effective task failure notification Key: FLINK-14375 URL: https://issues.apache.org/jira/browse/FLINK-14375 Project: Flink

[jira] [Created] (FLINK-14374) Enable RegionFailoverITCase to pass with scheduler NG

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14374: --- Summary: Enable RegionFailoverITCase to pass with scheduler NG Key: FLINK-14374 URL: https://issues.apache.org/jira/browse/FLINK-14374 Project: Flink Issue Type: Sub-t

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-11 Thread Zili Chen
Hi Kostas, Thanks for your reply. (1) cancelWithSavepoint() has already been excluded from the FLIP. But to emphasize that we make the decision to exclude it I add it to reject alternatives. (2) Updated FLIP to reflect the consensus :-) Best, tison. Kostas Kloudas 于2019年10月11日周五 下午5:12写道: >

[PROPOSAL] Contribute Stateful Functions to Apache Flink

2019-10-11 Thread Stephan Ewen
Dear Flink Community! Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**. Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk) that can communic

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

2019-10-11 Thread Timo Walther
Hi Jark, great to see that there is consensus on something like `CREATE FUNCTION bf1 WITH (precision=0.001)`. In the near future, there should be no difference between a catalog function and a temporary function. This should simplify the stack a lot. How about we postpone this topic after F

[jira] [Created] (FLINK-14373) Enable ZooKeeperHighAvailabilityITCase to pass with scheduler NG

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14373: --- Summary: Enable ZooKeeperHighAvailabilityITCase to pass with scheduler NG Key: FLINK-14373 URL: https://issues.apache.org/jira/browse/FLINK-14373 Project: Flink Issue

[jira] [Created] (FLINK-14372) Enable KeyedStateCheckpointingITCase to pass with scheduler NG

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14372: --- Summary: Enable KeyedStateCheckpointingITCase to pass with scheduler NG Key: FLINK-14372 URL: https://issues.apache.org/jira/browse/FLINK-14372 Project: Flink Issue T

[jira] [Created] (FLINK-14371) Enable ClassLoaderITCase to pass with scheduler NG

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14371: --- Summary: Enable ClassLoaderITCase to pass with scheduler NG Key: FLINK-14371 URL: https://issues.apache.org/jira/browse/FLINK-14371 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-11 Thread Kostas Kloudas
Hi all, I only have two minor comments before voting and they have to do with the following: 1) In the discussion, we agreed to remove the cancelWithSavepoint() from the JobClient as this is deprecated in the rest API. This is not in the FLIP. 2) The section "ClusterDescriptor or Executor(FLIP-73

Re: [VOTE] FLIP-74: Flink JobClient API

2019-10-11 Thread Biao Liu
+1 (non-binding), glad to have this improvement! Thanks, Biao /'bɪ.aʊ/ On Fri, 11 Oct 2019 at 14:44, Jeff Zhang wrote: > +1, overall design make sense to me > > SHI Xiaogang 于2019年10月11日周五 上午11:15写道: > > > +1. The interface looks fine to me. > > > > Regards, > > Xiaogang > > > > Zili Chen 于

[jira] [Created] (FLINK-14370) KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis

2019-10-11 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14370: - Summary: KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis Key: FLINK-14370 URL: https://issues.apache.org/jira/browse/FLI

[jira] [Created] (FLINK-14369) KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on Travis

2019-10-11 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14369: - Summary: KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on Travis Key: FLINK-14369 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (FLINK-14368) KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testCustomPartitioning failed on Travis

2019-10-11 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14368: - Summary: KafkaProducerAtLeastOnceITCase>KafkaProducerTestBase.testCustomPartitioning failed on Travis Key: FLINK-14368 URL: https://issues.apache.org/jira/browse/FLINK-14368

[jira] [Created] (FLINK-14367) Fully support for converting RexNode to Expression

2019-10-11 Thread Jark Wu (Jira)
Jark Wu created FLINK-14367: --- Summary: Fully support for converting RexNode to Expression Key: FLINK-14367 URL: https://issues.apache.org/jira/browse/FLINK-14367 Project: Flink Issue Type: Improvem

[DISCUSS] FLIP-77: Introduce ConfigOptions with Data Types

2019-10-11 Thread Timo Walther
Hi everyone, as mentioned in the [DISCUSS] thread of FLIP-54 [1]. The evolution of ConfigOption is a very sensitive topic that touches many implementers. We will therefore discard FLIP-54 and split it into several discussions and FLIPs. We propose FLIP-77 for introducing ConfigOptions with D

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

2019-10-11 Thread Jark Wu
Hi Timo, The reason to put the parameter in constructor instead of `eval` is that 1) the initialization for the parameters may take long 2) there might be a lot of parameters, it's verbose to specify them in every where it's used. Having some syntax like `CREATE FUNCTION bf1 WITH (precision=0.001

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

2019-10-11 Thread Chesnay Schepler
h I like job-/cluster partitions. On 10/10/2019 16:27, Till Rohrmann wrote: I think we should introduce a separate interface for the ResourceManager so that it can list and delete global result partitions from the shuffle service implementation. As long as the JM and RM run in the same proce

Re: [DISCUSS] FLIP-68: Extend Core Table System with Modular Plugins

2019-10-11 Thread Timo Walther
I'm fine with `type` for consistency reasons for now. I hope we will fix that when we rework the properties design. @Bowen: could you update the wiki page? I think we could start a vote, right? Regards, Timo On 11.10.19 04:31, Jark Wu wrote: Hi Timo, I agree that we are going to rework pro

[jira] [Created] (FLINK-14366) Annotate MiniCluster tests in flink-tests with AlsoRunWithSchedulerNG

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14366: --- Summary: Annotate MiniCluster tests in flink-tests with AlsoRunWithSchedulerNG Key: FLINK-14366 URL: https://issues.apache.org/jira/browse/FLINK-14366 Project: Flink

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

2019-10-11 Thread Timo Walther
Thanks for your feedback Kurt and Jark. I'm wondering why we allow setting `val bf1 = new BloomFilter(0.001)`, `val bf2 = new BloomFilter(0.005)`. This is not very SQL-like. Parameters should be passed when calling the function instead. Users can use the query itself to parametize the functio

[jira] [Created] (FLINK-14365) Annotate MiniCluster tests in core modules with AlsoRunWithSchedulerNG

2019-10-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14365: --- Summary: Annotate MiniCluster tests in core modules with AlsoRunWithSchedulerNG Key: FLINK-14365 URL: https://issues.apache.org/jira/browse/FLINK-14365 Project: Flink