Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Jark Wu
Thanks Hequn for the reviewing and merging. I'm preparing the RC1 for 1.9.1. Cheers, Jark On Mon, 30 Sep 2019 at 11:59, Hequn Cheng wrote: > Hi, > > @jincheng sun @Dian Fu > Thanks > a lot for reporting and fixing the problem! > @Jark Wu The PR of FLINK-14288 has been merged into > both ma

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

2019-09-29 Thread Becket Qin
Forgot to say that I agree with Till that it seems a good idea to let TEs register the global partitions to the RM instead of letting JM do it. This simplifies quite a few things. Thanks, Jiangjie (Becket) Qin On Sun, Sep 29, 2019 at 11:25 PM Becket Qin wrote: > Hi Chesnay, > > Thanks for the

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

2019-09-29 Thread Becket Qin
Hi Chesnay, Thanks for the proposal. My understanding of the entire workflow step by step is following: - JM maintains the local and global partition metadata when the task runs to create result partitions. The tasks themselves does not distinguish between local / global partitions. Only the J

Re: [DISCUSS] FLIP-74: Flink JobClient API

2019-09-29 Thread Konstantin Knauf
Hi Thomas, maybe there is a misunderstanding. There is no plan to deprecate anything in the REST API in the process of introducing the JobClient API, and it shouldn't. Since "cancel with savepoint" was already deprecated in the REST API and CLI, I am just raising the question whether to add it to

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Hequn Cheng
Hi, @jincheng sun @Dian Fu Thanks a lot for reporting and fixing the problem! @Jark Wu The PR of FLINK-14288 has been merged into both master and release-1.9. Looking forward to the first RC of 1.9.1. Best, Hequn On Mon, Sep 30, 2019 at 10:54 AM Jark Wu wrote: > Thanks Jincheng and Dian for

Re: [DISCUSS] Best practice to run flink on kubernetes

2019-09-29 Thread Yang Wang
Hi dev and users, I just want to revive this discussion because we have some meaningful progress about kubernetes native integration. I have made a draft implementation to complete the poc. Cli and submission are both working as expected. The design doc[1] has been updated, including the detailed

[jira] [Created] (FLINK-14289) Remove Optional fields from RecordWriter relevant classes

2019-09-29 Thread zhijiang (Jira)
zhijiang created FLINK-14289: Summary: Remove Optional fields from RecordWriter relevant classes Key: FLINK-14289 URL: https://issues.apache.org/jira/browse/FLINK-14289 Project: Flink Issue Type:

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Jark Wu
Thanks Jincheng and Dian for reporting and fixing this. Please let me know when FLINK-14288 is merged. Best, Jark On Mon, 30 Sep 2019 at 09:29, jincheng sun wrote: > Yes, Thanks for the quick response @Dian Fu . > > Dian Fu 于2019年9月30日周一 上午9:22写道: > > > Hi Jincheng, > > > > Thanks a lot for r

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread jincheng sun
Yes, Thanks for the quick response @Dian Fu . Dian Fu 于2019年9月30日周一 上午9:22写道: > Hi Jincheng, > > Thanks a lot for reporting this issue. Good catch! I'd like to take this > issue and fix it ASAP. Would you please assign it to me? > > Regards, > Dian > > On Mon, Sep 30, 2019 at 9:18 AM jincheng su

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Dian Fu
Hi Jincheng, Thanks a lot for reporting this issue. Good catch! I'd like to take this issue and fix it ASAP. Would you please assign it to me? Regards, Dian On Mon, Sep 30, 2019 at 9:18 AM jincheng sun wrote: > Hi Jark, > > I just found that there is NOTICE problem for source release, we shoul

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread jincheng sun
Hi Jark, I just found that there is NOTICE problem for source release, we should fix it in 1.9.1. The detail can be found in FLINK-14288. Best, Jincheng Debasish Ghosh 于2019年9月29日周日 下午4:14写道: > Thanks a lot! > > On Sun, Sep 29, 2019 at 1:42 PM Jark Wu wrote: > > > Hi Debasish, > > > > Yes. T

[jira] [Created] (FLINK-14288) Add Py4j NOTICE for source release

2019-09-29 Thread sunjincheng (Jira)
sunjincheng created FLINK-14288: --- Summary: Add Py4j NOTICE for source release Key: FLINK-14288 URL: https://issues.apache.org/jira/browse/FLINK-14288 Project: Flink Issue Type: Bug C

Re: [DISCUSS] FLIP-74: Flink JobClient API

2019-09-29 Thread Thomas Weise
I did not realize there was a plan to deprecate anything in the REST API? The REST API is super important for tooling written in non JVM languages, that does not include a Flink client (like FlinkK8sOperator). The REST API should continue to support all job management operations, including job sub

Re: [DISCUSS] FLIP-74: Flink JobClient API

2019-09-29 Thread Konstantin Knauf
Hi Zili, thanks for working on this topic. Just read through the FLIP and I have two questions: * should we add "cancelWithSavepeoint" to a new public API, when we have deprecated the corresponding REST API/CLI methods? In my understanding there is no reason to use it anymore. * should we call "s

[ANNOUNCE] Weekly Community Update 2019/39

2019-09-29 Thread Konstantin Knauf
Dear community, happy to share this week's community update with news about Flink 1.10 & 1.9.1, two FLIPs for better programmatic job and cluster control, improvements to the web user interface and a bit more. Flink Development == * [releases] Jark has started a discussion on releasi

Re: REST API / JarRunHandler: More flexibility for launching jobs

2019-09-29 Thread Zili Chen
Aljoscha & Thomas, The Program way won't work without exposing pipeline compiler supports because otherwise there is no way user creates a Pipeline by himself. I am not sure what it means above "running a job without a Java Client", if it means the server/cluster is Dispatcher, then the other lan

[jira] [Created] (FLINK-14287) Decouple leader address from LeaderContender

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14287: - Summary: Decouple leader address from LeaderContender Key: FLINK-14287 URL: https://issues.apache.org/jira/browse/FLINK-14287 Project: Flink Issue Type: Im

[jira] [Created] (FLINK-14286) Remove Akka specific parsing logic from LeaderConnectionInfo

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14286: - Summary: Remove Akka specific parsing logic from LeaderConnectionInfo Key: FLINK-14286 URL: https://issues.apache.org/jira/browse/FLINK-14286 Project: Flink

[jira] [Created] (FLINK-14285) Simplify Dispatcher factories by removing generics

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14285: - Summary: Simplify Dispatcher factories by removing generics Key: FLINK-14285 URL: https://issues.apache.org/jira/browse/FLINK-14285 Project: Flink Issue Ty

[jira] [Created] (FLINK-14284) Add shut down future to Dispatcher

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14284: - Summary: Add shut down future to Dispatcher Key: FLINK-14284 URL: https://issues.apache.org/jira/browse/FLINK-14284 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14283) Update Kinesis consumer documentation for watermarks and event time alignment

2019-09-29 Thread Thomas Weise (Jira)
Thomas Weise created FLINK-14283: Summary: Update Kinesis consumer documentation for watermarks and event time alignment Key: FLINK-14283 URL: https://issues.apache.org/jira/browse/FLINK-14283 Project

[jira] [Created] (FLINK-14282) Simplify DispatcherResourceManagerComponent

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14282: - Summary: Simplify DispatcherResourceManagerComponent Key: FLINK-14282 URL: https://issues.apache.org/jira/browse/FLINK-14282 Project: Flink Issue Type: Imp

[jira] [Created] (FLINK-14281) Introduce DispatcherRunner#getShutDownFuture

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14281: - Summary: Introduce DispatcherRunner#getShutDownFuture Key: FLINK-14281 URL: https://issues.apache.org/jira/browse/FLINK-14281 Project: Flink Issue Type: Im

[jira] [Created] (FLINK-14280) Introduce DispatcherRunner for better separation of concerns

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14280: - Summary: Introduce DispatcherRunner for better separation of concerns Key: FLINK-14280 URL: https://issues.apache.org/jira/browse/FLINK-14280 Project: Flink

[jira] [Created] (FLINK-14279) The Table API code walkthrough should be translated into chinese.

2019-09-29 Thread gaofeilong (Jira)
gaofeilong created FLINK-14279: -- Summary: The Table API code walkthrough should be translated into chinese. Key: FLINK-14279 URL: https://issues.apache.org/jira/browse/FLINK-14279 Project: Flink

[jira] [Created] (FLINK-14278) Pass in ioExecutor into AbstractDispatcherResourceManagerComponentFactory

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14278: - Summary: Pass in ioExecutor into AbstractDispatcherResourceManagerComponentFactory Key: FLINK-14278 URL: https://issues.apache.org/jira/browse/FLINK-14278 Project:

[jira] [Created] (FLINK-14277) Replace InMemoryJobGraphStore with proper TestingJobGraphStore implementation

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14277: - Summary: Replace InMemoryJobGraphStore with proper TestingJobGraphStore implementation Key: FLINK-14277 URL: https://issues.apache.org/jira/browse/FLINK-14277 Proje

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

2019-09-29 Thread Xintong Song
Thanks for drafting the FLIP and starting this discussion, Yadong. I have some comments: - I can see that the proposed memory and cpu usage to be displayed (in section 1.1) are aligned with the current ResourceProfile fields. However, we are working on changing the memory fields in 1.1

[jira] [Created] (FLINK-14276) Quickstarts Scala nightly end-to-end test fails on Travis

2019-09-29 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-14276: - Summary: Quickstarts Scala nightly end-to-end test fails on Travis Key: FLINK-14276 URL: https://issues.apache.org/jira/browse/FLINK-14276 Project: Flink I

[jira] [Created] (FLINK-14275) Window Aggregate function support a RichFunction

2019-09-29 Thread hailong wang (Jira)
hailong wang created FLINK-14275: Summary: Window Aggregate function support a RichFunction Key: FLINK-14275 URL: https://issues.apache.org/jira/browse/FLINK-14275 Project: Flink Issue Type:

[jira] [Created] (FLINK-14274) RuntimeContext can be get from FunctionContext

2019-09-29 Thread hailong wang (Jira)
hailong wang created FLINK-14274: Summary: RuntimeContext can be get from FunctionContext Key: FLINK-14274 URL: https://issues.apache.org/jira/browse/FLINK-14274 Project: Flink Issue Type: I

[jira] [Created] (FLINK-14273) when User-Defined Aggregate Functions(UDAF) parameters are inconsistent with the definition, the error reporting is confusing

2019-09-29 Thread hailong wang (Jira)
hailong wang created FLINK-14273: Summary: when User-Defined Aggregate Functions(UDAF) parameters are inconsistent with the definition, the error reporting is confusing Key: FLINK-14273 URL: https://issues.apache.

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Debasish Ghosh
Thanks a lot .. On Sun, Sep 29, 2019 at 1:36 PM Jark Wu wrote: > Hi Debasish, > > +1 have FLINK-12501 in 1.9.1. As there is nobody against it under the JIRA > issue. I just cherry-picked it to 1.9 branch. > > Best, > Jark > > On Sun, 29 Sep 2019 at 13:09, Debasish Ghosh > wrote: > >> Hi - >> >>

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Debasish Ghosh
Thanks a lot! On Sun, Sep 29, 2019 at 1:42 PM Jark Wu wrote: > Hi Debasish, > > Yes. The maven artifacts will also be updated. I can upgrade the version of > flink-avro to 1.9.1 once it is released. > > Cheers, > Jark > > On Sun, 29 Sep 2019 at 16:08, Debasish Ghosh > wrote: > > > 1 more questi

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Jark Wu
Hi Debasish, Yes. The maven artifacts will also be updated. I can upgrade the version of flink-avro to 1.9.1 once it is released. Cheers, Jark On Sun, 29 Sep 2019 at 16:08, Debasish Ghosh wrote: > 1 more question .. since it has been cherry picked to 1.9 branch will the > maven artifacts of 1.

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Debasish Ghosh
1 more question .. since it has been cherry picked to 1.9 branch will the maven artifacts of 1.9 also be updated accordingly ? regards. On Sun, Sep 29, 2019 at 1:37 PM Debasish Ghosh wrote: > Thanks a lot .. > > On Sun, Sep 29, 2019 at 1:36 PM Jark Wu wrote: > >> Hi Debasish, >> >> +1 have FLI

Re: [DISCUSS] Releasing Flink 1.9.1

2019-09-29 Thread Jark Wu
Hi Debasish, +1 have FLINK-12501 in 1.9.1. As there is nobody against it under the JIRA issue. I just cherry-picked it to 1.9 branch. Best, Jark On Sun, 29 Sep 2019 at 13:09, Debasish Ghosh wrote: > Hi - > > Will FLINK-12501 be included in 1.9.1 ? > > regards > > On Sun, 29 Sep 2019 at 9:04 AM

[jira] [Created] (FLINK-14272) Support Blink planner for Python UDF

2019-09-29 Thread Dian Fu (Jira)
Dian Fu created FLINK-14272: --- Summary: Support Blink planner for Python UDF Key: FLINK-14272 URL: https://issues.apache.org/jira/browse/FLINK-14272 Project: Flink Issue Type: Sub-task Com

[jira] [Created] (FLINK-14271) Deprecate legacy RestartPipelinedRegionStrategy

2019-09-29 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14271: --- Summary: Deprecate legacy RestartPipelinedRegionStrategy Key: FLINK-14271 URL: https://issues.apache.org/jira/browse/FLINK-14271 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-14270) new web ui should display more than 4 metrics

2019-09-29 Thread David Anderson (Jira)
David Anderson created FLINK-14270: -- Summary: new web ui should display more than 4 metrics Key: FLINK-14270 URL: https://issues.apache.org/jira/browse/FLINK-14270 Project: Flink Issue Type:

[jira] [Created] (FLINK-14269) The close method does not work for UserDefinedFunction

2019-09-29 Thread Dian Fu (Jira)
Dian Fu created FLINK-14269: --- Summary: The close method does not work for UserDefinedFunction Key: FLINK-14269 URL: https://issues.apache.org/jira/browse/FLINK-14269 Project: Flink Issue Type: Sub-