Re: [DISCUSS] Release 1.5.5 and 1.6.2

2018-10-15 Thread Chesnay Schepler
My issues have been merged as well. I will cut the release branches in 3 hours. On 15.10.2018 22:11, Till Rohrmann wrote: FLINK-9932 has been merged. Cheers, Till On Mon, Oct 15, 2018 at 1:19 PM Till Rohrmann wrote: Thanks a lot for starting this discussion Chesnay. I fully agree that a ne

[jira] [Created] (FLINK-10560) JVM_ARGS doesn't work in jobmanager and taskmangers' JVM on yarn

2018-10-15 Thread Jiayi Liao (JIRA)
Jiayi Liao created FLINK-10560: -- Summary: JVM_ARGS doesn't work in jobmanager and taskmangers' JVM on yarn Key: FLINK-10560 URL: https://issues.apache.org/jira/browse/FLINK-10560 Project: Flink

Re: Creating a slide set for a Flink intro talk

2018-10-15 Thread Hequn Cheng
Hi @Fabian Thanks a lot for creating the slide. It is great to have an introduction slide to our users. I have taken a look at the slide. The content of the slide is good and I only left some minor suggestions. Best, Hequn On Sat, Oct 13, 2018 at 10:12 AM jincheng sun wrote: > @Fabian Hueske

Re: Add checkpoint hooks to window triggers

2018-10-15 Thread Paul Lam
Hi Fabian, Thanks for your reply! It takes me a while to reconsider the proposal, and I think you’re right. The checkpoint hook would affect a lot of mechanisms we already have, and it’s unworthy. > Wouldn't a transactional sink provide exactly the same guarantees? Yes, it is. But it needs to

[jira] [Created] (FLINK-10559) Remove LegacyLocalStreamEnvironment

2018-10-15 Thread TisonKun (JIRA)
TisonKun created FLINK-10559: Summary: Remove LegacyLocalStreamEnvironment Key: FLINK-10559 URL: https://issues.apache.org/jira/browse/FLINK-10559 Project: Flink Issue Type: Sub-task Co

[jira] [Created] (FLINK-10558) Port YARNHighAvailabilityITCase and YARNSessionCapacitySchedulerITCase to new code base

2018-10-15 Thread vinoyang (JIRA)
vinoyang created FLINK-10558: Summary: Port YARNHighAvailabilityITCase and YARNSessionCapacitySchedulerITCase to new code base Key: FLINK-10558 URL: https://issues.apache.org/jira/browse/FLINK-10558 Proje

[jira] [Created] (FLINK-10557) Checkpoint size metric incorrectly reports the same value until restart

2018-10-15 Thread Ruby Andrews (JIRA)
Ruby Andrews created FLINK-10557: Summary: Checkpoint size metric incorrectly reports the same value until restart Key: FLINK-10557 URL: https://issues.apache.org/jira/browse/FLINK-10557 Project: Flin

Re: [DISCUSS] Release 1.5.5 and 1.6.2

2018-10-15 Thread Till Rohrmann
FLINK-9932 has been merged. Cheers, Till On Mon, Oct 15, 2018 at 1:19 PM Till Rohrmann wrote: > Thanks a lot for starting this discussion Chesnay. I fully agree that a > new bug fix release would be justified. > > I'm currently working on FLINK-9932 which I would like to include in the > next b

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-15 Thread Zhang, Xuefu
Hi Shuyi, Thank you for your input. Yes, I agreed with a phased approach and like to move forward fast. :) We did some work internally on DDL utilizing babel parser in Calcite. While babel makes Calcite's grammar extensible, at first impression it still seems too cumbersome for a project when t

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-15 Thread Zhang, Xuefu
Hi Bowen, Thank you for your feedback and interest in the project. Your contribution is certainly welcome. Per your suggestion, I have created an Uber JIRA (https://issues.apache.org/jira/browse/FLINK-10556) to track our overall effort on this. For each subtask, we'd like to see a short descrip

[jira] [Created] (FLINK-10556) Integration with Apache Hive

2018-10-15 Thread Xuefu Zhang (JIRA)
Xuefu Zhang created FLINK-10556: --- Summary: Integration with Apache Hive Key: FLINK-10556 URL: https://issues.apache.org/jira/browse/FLINK-10556 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-10555) Port AkkaSslITCase to new code base

2018-10-15 Thread TisonKun (JIRA)
TisonKun created FLINK-10555: Summary: Port AkkaSslITCase to new code base Key: FLINK-10555 URL: https://issues.apache.org/jira/browse/FLINK-10555 Project: Flink Issue Type: Sub-task Co

[jira] [Created] (FLINK-10554) Bump flink-shaded dependency version

2018-10-15 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10554: Summary: Bump flink-shaded dependency version Key: FLINK-10554 URL: https://issues.apache.org/jira/browse/FLINK-10554 Project: Flink Issue Type: Impr

[jira] [Created] (FLINK-10553) Unified table sink and source name in SQL statement

2018-10-15 Thread wangsan (JIRA)
wangsan created FLINK-10553: --- Summary: Unified table sink and source name in SQL statement Key: FLINK-10553 URL: https://issues.apache.org/jira/browse/FLINK-10553 Project: Flink Issue Type: Bug

[RESULT][VOTE] Release flink-shaded 5.0, release candidate #1

2018-10-15 Thread Chesnay Schepler
|I'm happy to announce that we have unanimously approved this release.| |There are 4 approving votes, 3 of which are binding:| |* Aljoscha (binding) | |* Timo (binding) | |* Gordin (binding) | |* vino (non-binding) | |There are no disapproving votes.| |Thanks everyone!| On 10.10.2018 15:11, Ches

Re: Add checkpoint hooks to window triggers

2018-10-15 Thread Fabian Hueske
Hi Paul, I think this would be very tricky to implement and interfere with many parts of the system like state backends, checkpointing logic, etc. We would need to maintain a copy (or version) of the state at the time of a checkpoint. There might be multiple checkpoints in flight. Checkpoints migh

Re: Add checkpoint hooks to window triggers

2018-10-15 Thread Paul Lam
Hi Fabian, Perhaps I didn’t explain that clearly. Actually I want a trigger to fire when a checkpoint is completed, and emit the intermediate results in consistency with the completed checkpoint. It works like this: 1) Once the window operator receives a barrier, it performs the snapshot as us

Re: [DISCUSS] Release 1.5.5 and 1.6.2

2018-10-15 Thread Till Rohrmann
Thanks a lot for starting this discussion Chesnay. I fully agree that a new bug fix release would be justified. I'm currently working on FLINK-9932 which I would like to include in the next bug fix release. It should be done by the end of today. Cheers, Till On Mon, Oct 15, 2018 at 11:40 AM Ches

回复:[DISCUSS] Improve broadcast serialization

2018-10-15 Thread Zhijiang(wangzhijiang999)
Let's come back to this discussion again. Thanks for @Nico Kruber and @Piotr Nowojski reviewing the PR of proposed 2.1 for serialization only once below and it is already merged into branch. For proposed 2.2 for copy only once, we verified it is also very important for batch job related with j

[DISCUSS] Release 1.5.5 and 1.6.2

2018-10-15 Thread Chesnay Schepler
Hello, we've accumulated various fixes for 1.5.5 (24) and 1.6.2 (37) that improve stability by quite a bit along with some neat usability fixes. I'm proposing to do the next bugfix releases soon (I suggest tomorrow as a tentative vote date), and volunteer to handle the release process for bo

Re: Please review : Re: PubSub connector (FLINK-9311)

2018-10-15 Thread Richard Deurwaarder
Hello everyone, To improve performance we have changed some parts of the code: * Previously the same threads used to poll PubSub we're used to actually run the flink pipeline, this caused some issues in the PubSub SDK. We've changed this so that the PubSub threads are only used to poll PubSub and

Re: Handling burst I/O when using tumbling/sliding windows

2018-10-15 Thread Piotr Nowojski
Hi, Just to sum up this thread. Discussion in the before mentioned design doc concluded that there is no need for API changes. The semantic that Rong was asking for can be achieved by implementing a custom WindowAssigner, that mimics one of the existing ones like TumblingEventTimeWindows, and u

[jira] [Created] (FLINK-10552) Provide RichAsyncFunction for scala API

2018-10-15 Thread Shimin Yang (JIRA)
Shimin Yang created FLINK-10552: --- Summary: Provide RichAsyncFunction for scala API Key: FLINK-10552 URL: https://issues.apache.org/jira/browse/FLINK-10552 Project: Flink Issue Type: Bug

Re: Add checkpoint hooks to window triggers

2018-10-15 Thread Fabian Hueske
Hi Paul, If I got your proposal right, you'd like to fire a Trigger right before a checkpoint is taken, correct? So, before taking a checkpoint, a Trigger would fire and the operator would process and emit some intermediate results. This approach would not completely solve the consistency issue b

[jira] [Created] (FLINK-10551) Remove legacy REST handlers

2018-10-15 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10551: Summary: Remove legacy REST handlers Key: FLINK-10551 URL: https://issues.apache.org/jira/browse/FLINK-10551 Project: Flink Issue Type: Sub-task