Re: [DISCUSS] Proposal of external shuffle service

2018-10-31 Thread Jin Sun
Thanks Zhijiang for the proposal. I like the idea of external shuffle service, have left some comments on the document. > On Oct 31, 2018, at 2:26 AM, Till Rohrmann wrote: > > Thanks for the update Zhijiang! The community is currently quite busy with > the next Flink release. I hope that we ca

[jira] [Created] (FLINK-10744) Integrate Flink with Hive metastore

2018-10-31 Thread Xuefu Zhang (JIRA)
Xuefu Zhang created FLINK-10744: --- Summary: Integrate Flink with Hive metastore Key: FLINK-10744 URL: https://issues.apache.org/jira/browse/FLINK-10744 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-10743) Use 0 processExitCode for ApplicationStatus.CANCELED

2018-10-31 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-10743: --- Summary: Use 0 processExitCode for ApplicationStatus.CANCELED Key: FLINK-10743 URL: https://issues.apache.org/jira/browse/FLINK-10743 Project: Flink Issue Type

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-31 Thread Zhang, Xuefu
Hi Shuiyi, Good idea. Actually the PDF was converted from a google doc. Here is its link: https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit?usp=sharing Once we reach an agreement, I can convert it to a FLIP. Thanks, Xuefu

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-31 Thread Shuyi Chen
Hi Xuefu, Thanks a lot for driving this big effort. I would suggest convert your proposal and design doc into a google doc, and share it on the dev mailing list for the community to review and comment with title like "[DISCUSS] ... Hive integration design ..." . Once approved, we can document it

Re: Sharing state between subtasks

2018-10-31 Thread Thomas Weise
Hi, We are planning to work on the Kinesis consumer in the following order: 1. Add per shard watermarking: https://issues.apache.org/jira/browse/FLINK-5697 - this will be code we already use internally; I will open a PR to add it to the Flink Kinesis consumer 2. Exchange of per subtask watermarks

[ANNOUNCE] Feature freeze for Flink 1.7

2018-10-31 Thread Till Rohrmann
Dear community, hereby I announce the official feature freeze for Flink 1.7. In order to ease the testing and bug fixing, I have not cut a release branch yet. This, however, requires that no more features will be merged until we create the release branch. My plan is to cut the release branch at t

[jira] [Created] (FLINK-10742) Let Netty use Flink's buffers directly in credit-based mode

2018-10-31 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-10742: --- Summary: Let Netty use Flink's buffers directly in credit-based mode Key: FLINK-10742 URL: https://issues.apache.org/jira/browse/FLINK-10742 Project: Flink Is

[jira] [Created] (FLINK-10741) GraphOperationsITCase.testFilterOnVerticesSugar deadlocks on Travis

2018-10-31 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-10741: - Summary: GraphOperationsITCase.testFilterOnVerticesSugar deadlocks on Travis Key: FLINK-10741 URL: https://issues.apache.org/jira/browse/FLINK-10741 Project: Flink

[DISCUSS] FLIP-27: Refactor Source Interface

2018-10-31 Thread Aljoscha Krettek
Hi All, In order to finally get the ball rolling on the new source interface that we have discussed for so long I finally created a FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface I cc'ed Thomas and Jamie because of the ongoing work/discussion about

[jira] [Created] (FLINK-10740) Refactor Source Interface

2018-10-31 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created FLINK-10740: Summary: Refactor Source Interface Key: FLINK-10740 URL: https://issues.apache.org/jira/browse/FLINK-10740 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-10739) Unstable ProcessFailureCancelingITCase.testCancelingOnProcessFailure

2018-10-31 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-10739: -- Summary: Unstable ProcessFailureCancelingITCase.testCancelingOnProcessFailure Key: FLINK-10739 URL: https://issues.apache.org/jira/browse/FLINK-10739 Project: Fli

[jira] [Created] (FLINK-10738) FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed on Travis

2018-10-31 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-10738: - Summary: FlinkKafkaProducerITCase.testScaleUpAfterScalingDown failed on Travis Key: FLINK-10738 URL: https://issues.apache.org/jira/browse/FLINK-10738 Project: Flin

[jira] [Created] (FLINK-10737) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis

2018-10-31 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-10737: - Summary: FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint failed on Travis Key: FLINK-10737 URL: https://issues.apache.org/jira/browse/FLINK-10737 Projec

[jira] [Created] (FLINK-10736) Shaded Hadoop S3A end-to-end test failed on Travis

2018-10-31 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-10736: - Summary: Shaded Hadoop S3A end-to-end test failed on Travis Key: FLINK-10736 URL: https://issues.apache.org/jira/browse/FLINK-10736 Project: Flink Issue Ty

[jira] [Created] (FLINK-10735) flink on yarn close container exception

2018-10-31 Thread Fei Feng (JIRA)
Fei Feng created FLINK-10735: Summary: flink on yarn close container exception Key: FLINK-10735 URL: https://issues.apache.org/jira/browse/FLINK-10735 Project: Flink Issue Type: Bug Com

[jira] [Created] (FLINK-10734) Temporal joins on heavily filtered tables might fail in planning

2018-10-31 Thread Piotr Nowojski (JIRA)
Piotr Nowojski created FLINK-10734: -- Summary: Temporal joins on heavily filtered tables might fail in planning Key: FLINK-10734 URL: https://issues.apache.org/jira/browse/FLINK-10734 Project: Flink

[jira] [Created] (FLINK-10733) Misleading clean_log_files() in common.sh

2018-10-31 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-10733: -- Summary: Misleading clean_log_files() in common.sh Key: FLINK-10733 URL: https://issues.apache.org/jira/browse/FLINK-10733 Project: Flink Issue Type: Bug

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

2018-10-31 Thread Till Rohrmann
I agree with Chesnay that the web UI is already fairly isolated in Flink's code base (see flink-runtime-web module). Moreover, imo it is very important that Flink comes with a nice web UI which works out of the box because it lowers the entrance hurdle massively. What one could argue is to make it

Re: FLINK-9752 (s3 recoverable writer) not actually fixed in 1.6.2?

2018-10-31 Thread Kostas Kloudas
Hi Addison, This is definitely an error on my end, as the feature is going to be available from Flink 1.7 onwards. I forgot to correctly update the Flink version when closing the JIRA issue. I will update the release notes accordingly. Sorry for the miscommunication, Kostas > On Oct 31, 2018,

Re: [DISCUSS] Proposal of external shuffle service

2018-10-31 Thread Till Rohrmann
Thanks for the update Zhijiang! The community is currently quite busy with the next Flink release. I hope that we can finish the release in two weeks. After that people will become more responsive again. Cheers, Till On Wed, Oct 31, 2018 at 7:49 AM zhijiang wrote: > I already created the umbrel

Re: Run Flink Natively On Kubernetes

2018-10-31 Thread Till Rohrmann
Thanks for sharing this design document with the community. I really like the idea of having a better K8s integration in Flink. This will allow people to use Flink more easily in different environments. Cheers, Till On Wed, Oct 31, 2018 at 1:55 AM Jin Sun wrote: > Hi all, > > FLIP-6 outlines im

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

2018-10-31 Thread Chesnay Schepler
The WebUI goes entirely through the REST API. I don't see how moving it out would make it easier to develop/contribute. The UI is already encapsulated in it's own module (flink-runtime-web) and changes can be made to it independently. Historically the issue was simply that contributions to the

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

2018-10-31 Thread Robert Metzger
Hey Fabian, First of all, big +1 on the effort! Two quick thoughts: - I would suggest to build a new UI outside of the main Flink project, and look at replacing the current one, once a new one has reached feature parity. - I would also suggest to go with React, as it seems the leading frontend fr

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

2018-10-31 Thread Flavio Pompermaier
I think that it is important to have a nice "official" (or at least free) Flink UI, we use it to see the detail of the jobs. It's very useful for people starting working with Flink and also for those that does not have the resources to write a custom UI. How are you going to monitor the status of a

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

2018-10-31 Thread dyana . rose
Re: who's using the web ui Though many mature solutions, with a fair amount of time/resources available are likely running their own front ends, for teams like mine which are smaller and aren't focused solely on working with Flink, having the web ui available removes a large barrier to getting

[jira] [Created] (FLINK-10732) force-shading 1.5.5 maven artifact was not released

2018-10-31 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10732: Summary: force-shading 1.5.5 maven artifact was not released Key: FLINK-10732 URL: https://issues.apache.org/jira/browse/FLINK-10732 Project: Flink I

[jira] [Created] (FLINK-10731) Support AVG on Date fields

2018-10-31 Thread Flavio Pompermaier (JIRA)
Flavio Pompermaier created FLINK-10731: -- Summary: Support AVG on Date fields Key: FLINK-10731 URL: https://issues.apache.org/jira/browse/FLINK-10731 Project: Flink Issue Type: Improvemen

Re: [DISCUSS] Change underlying Frontend Architecture for Flink Web Dashboard

2018-10-31 Thread Fabian Wollert
Hi Till, I basically agree with all your points. i would stress the "dustiness" of the current architecture: the package manager used (bower) is deprecated since a long time, the chance for the builds of the flink web dashboard not working anymore is increasing every day. About the knowledge in th

[jira] [Created] (FLINK-10730) How to resume Flink SQL Job ?

2018-10-31 Thread xinzhang (JIRA)
xinzhang created FLINK-10730: Summary: How to resume Flink SQL Job ? Key: FLINK-10730 URL: https://issues.apache.org/jira/browse/FLINK-10730 Project: Flink Issue Type: Improvement Compo