Re: [DISCUSS] Proposal for Flink job execution/availability metrics impovement

2019-05-28 Thread Hwanju Kim
(Somehow my email has failed to be sent multiple times, so I am using my personal email account) Hi, Piotrek - Thanks for the feedback! I revised the doc as commented. Here's the second part about exception classification - https://docs.google.com/document/d/1pcHg9F3GoDDeVD5GIIo2wO67Hmjgy0-hRDeu

[jira] [Created] (FLINK-12665) Introduce MiniBatchIntervalInferRule and watermark assigner operators

2019-05-28 Thread godfrey he (JIRA)
godfrey he created FLINK-12665: -- Summary: Introduce MiniBatchIntervalInferRule and watermark assigner operators Key: FLINK-12665 URL: https://issues.apache.org/jira/browse/FLINK-12665 Project: Flink

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-28 Thread Nico Kruber
ok, thanks for the investigation/clarification On 28/05/2019 16:28, Chesnay Schepler wrote: > So from what I found jackson pre 2.8 was including a relocated version > of snakeyaml, without creating a dependency-reduced pom. > As a result we always bundled it twice, with the second one not being >

[jira] [Created] (FLINK-12664) Implement TableSink to write Hive tables

2019-05-28 Thread Rui Li (JIRA)
Rui Li created FLINK-12664: -- Summary: Implement TableSink to write Hive tables Key: FLINK-12664 URL: https://issues.apache.org/jira/browse/FLINK-12664 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-41: Unify Keyed State Snapshot Binary Format for Savepoints

2019-05-28 Thread Tzu-Li (Gordon) Tai
I also should point out something that I forgot to mention in the initial post: Stefan has helped a lot in understanding the current status of state backends and also participated a lot in design choices for the FLIP :) On Wed, May 29, 2019 at 5:02 AM Tzu-Li (Gordon) Tai wrote: > Hi Flink devs,

Re: [DISCUSS] FLIP-41: Unify Keyed State Snapshot Binary Format for Savepoints

2019-05-28 Thread Congxian Qiu
Hi Gordon Thanks for the nice work. Big +1 for this. With the proposal, we’ll have unified binary format for savepoints, so users can switch backend using savepoint if they needed. And Users will be able to seamlessly migrate from previous savepoints of older Flink versions as the wiki said. B

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread Congxian Qiu
Hi Jincheng, Thanks for starting the discussion. +1 for 1.8.1 [Blocker] [FLINK-12296 Data loss silently in RocksDBStateBackend when more than one]  has been resolved before(master, release-1.8, release-1.7, and release-1.6), but the status of the issue has not updated, I’ve closed the issue no

Re: [DISCUSS] Features for Apache Flink 1.9.0

2019-05-28 Thread Biao Liu
Thanks for being the release manager, Gordon & Kurt. For FLIP-27, there are still some more details need to discuss. I don't think it could catch up the release of 1.9. @Aljoscha, @Stephan, do you agree that? zhijiang 于2019年5月28日周二 下午11:28写道: > Hi Gordon, > > Thanks for the kind reminder of fea

[jira] [Created] (FLINK-12663) Implement HiveTableSource to read Hive tables

2019-05-28 Thread zjuwangg (JIRA)
zjuwangg created FLINK-12663: Summary: Implement HiveTableSource to read Hive tables Key: FLINK-12663 URL: https://issues.apache.org/jira/browse/FLINK-12663 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12662) Flink how jobs for failover in history server as well

2019-05-28 Thread Su Ralph (JIRA)
Su Ralph created FLINK-12662: Summary: Flink how jobs for failover in history server as well Key: FLINK-12662 URL: https://issues.apache.org/jira/browse/FLINK-12662 Project: Flink Issue Type: Imp

NPE in DataStream

2019-05-28 Thread Abderrazak TAZARNI
Hi, In my Flink cluster , i have continuously an NPE: java.lang.NullPointerException at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriters(StreamTask.java:1175) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:212) at org.apache.flink.streaming.runtime.

[jira] [Created] (FLINK-12661) test failure: RestAPIStabilityTest.testDispatcherRestAPIStability

2019-05-28 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12661: Summary: test failure: RestAPIStabilityTest.testDispatcherRestAPIStability Key: FLINK-12661 URL: https://issues.apache.org/jira/browse/FLINK-12661 Project: Flink Is

[DISCUSS] FLIP-41: Unify Keyed State Snapshot Binary Format for Savepoints

2019-05-28 Thread Tzu-Li (Gordon) Tai
Hi Flink devs, Congxian, Kostas, and I have recently been discussing to unify the binary formats for keyed state in savepoints, which would allow for more operational flexibility such as swapping state backends across restores. As part of this FLIP, another main proposal is to start allowing chec

job specific configs

2019-05-28 Thread Moorhead,Richard
Is there any existing sentiment towards the use case of user specified configuration specified at job submission time? Eg: flink run -conf file.conf myjar.jar arg1 arg2 CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the ad

Application for contributor permission

2019-05-28 Thread 杨钊
Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is IMC666.

[jira] [Created] (FLINK-12660) Integrate Flink with Hive GenericUDAFResolver and GenericUDAFResolver2

2019-05-28 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12660: Summary: Integrate Flink with Hive GenericUDAFResolver and GenericUDAFResolver2 Key: FLINK-12660 URL: https://issues.apache.org/jira/browse/FLINK-12660 Project: Flink

[jira] [Created] (FLINK-12659) Integrate Flink with Hive GenericUDTF

2019-05-28 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12659: Summary: Integrate Flink with Hive GenericUDTF Key: FLINK-12659 URL: https://issues.apache.org/jira/browse/FLINK-12659 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12658) Integrate Flink with Hive GenericUDF

2019-05-28 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12658: Summary: Integrate Flink with Hive GenericUDF Key: FLINK-12658 URL: https://issues.apache.org/jira/browse/FLINK-12658 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12657) Integrate Flink with Hive UDF

2019-05-28 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12657: Summary: Integrate Flink with Hive UDF Key: FLINK-12657 URL: https://issues.apache.org/jira/browse/FLINK-12657 Project: Flink Issue Type: Sub-task Report

[jira] [Created] (FLINK-12656) Integrate Flink with Hive user defined functions

2019-05-28 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12656: Summary: Integrate Flink with Hive user defined functions Key: FLINK-12656 URL: https://issues.apache.org/jira/browse/FLINK-12656 Project: Flink Issue Type: New Feat

[jira] [Created] (FLINK-12655) SlotCountExceedingParallelismTest failed on Travis

2019-05-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12655: Summary: SlotCountExceedingParallelismTest failed on Travis Key: FLINK-12655 URL: https://issues.apache.org/jira/browse/FLINK-12655 Project: Flink Is

Re: [DISCUSS] Features for Apache Flink 1.9.0

2019-05-28 Thread zhijiang
Hi Gordon, Thanks for the kind reminder of feature freeze date for 1.9.0. I think the date makes sense on my side. For FLIP-31, I and Andrey could be done within two weeks or so. And I already finished my side work for FLIP-1. Best, Zhijiang --

[jira] [Created] (FLINK-12654) Enable Azure FS E2E tests on Travis

2019-05-28 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-12654: - Summary: Enable Azure FS E2E tests on Travis Key: FLINK-12654 URL: https://issues.apache.org/jira/browse/FLINK-12654 Project: Flink Issue Type: Improvement

KeyGroup management, data locality

2019-05-28 Thread Smirnov Sergey Vladimirovich
Hello flink-dev, The aim is to achieve data locality for source connectors and for (dynamic for our case) task/operator scaling. Current version (flink 1.8) - some_hash(key_hash) % parallelism - just a round-robin algorithm. Proposal - control on datastream object-key layer its key-group distrib

[jira] [Created] (FLINK-12653) Keyed state backend fails to restore during rescaling

2019-05-28 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-12653: -- Summary: Keyed state backend fails to restore during rescaling Key: FLINK-12653 URL: https://issues.apache.org/jira/browse/FLINK-12653 Project: Flink

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-28 Thread Chesnay Schepler
So from what I found jackson pre 2.8 was including a relocated version of snakeyaml, without creating a dependency-reduced pom. As a result we always bundled it twice, with the second one not being used at all (I guess?) They have now removed the inclusion, meaning they now work against the sn

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-28 Thread Nico Kruber
this is a bit different here: flink-shaded-jackson-2.7.9-6.0/org/apache/flink/shaded/jackson2/com/fasterxml/jackson/dataformat/yaml/snakeyaml> find . . ./DumperOptions$FlowStyle.class ./DumperOptions$LineBreak.class ./DumperOptions$ScalarStyle.class ./DumperOptions$Version.class ./DumperOptions.cl

[jira] [Created] (FLINK-12652) Add Glossary to Concepts Section of Documentation

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12652: Summary: Add Glossary to Concepts Section of Documentation Key: FLINK-12652 URL: https://issues.apache.org/jira/browse/FLINK-12652 Project: Flink Iss

[jira] [Created] (FLINK-12651) Add "Style Guide" to Documentation Contribution Guide

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12651: Summary: Add "Style Guide" to Documentation Contribution Guide Key: FLINK-12651 URL: https://issues.apache.org/jira/browse/FLINK-12651 Project: Flink

[jira] [Created] (FLINK-12650) Redirect Users to Documentation Homepage if Requested Resource Does Not Exist

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12650: Summary: Redirect Users to Documentation Homepage if Requested Resource Does Not Exist Key: FLINK-12650 URL: https://issues.apache.org/jira/browse/FLINK-12650

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-28 Thread Chesnay Schepler
@Nico I still see snakeyaml in the flink-shaded-jackson jars, and in the dependency tree. It even takes up more space than before. AFAIK we do not work directly against the contained snakeyaml version, so we should be fine in any case. On 27/05/2019 16:45, Nico Kruber wrote: +1 (mod jackson sn

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-28 Thread Nico Kruber
+1 (mod jackson snakeyaml support; see below) - checked release notes - verified signatures (sha512, asc) - verified "mvn clean package -Pinclude-netty-tcnative-static" - compared contents and verified changes of + flink-shaded-asm-6*.jar + flink-shaded-guava-18.0*.jar + flink-shaded-netty-4

[jira] [Created] (FLINK-12649) Add a shim layer to support multiple versions of HMS

2019-05-28 Thread Rui Li (JIRA)
Rui Li created FLINK-12649: -- Summary: Add a shim layer to support multiple versions of HMS Key: FLINK-12649 URL: https://issues.apache.org/jira/browse/FLINK-12649 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Features for Apache Flink 1.9.0

2019-05-28 Thread Timo Walther
Thanks for being the release managers, Kurt and Gordon! From the Table & SQL API side, there are still a lot of open issues that need to be solved to decouple the API from a planner and enable the Blink planner. Also we need to make sure that the Blink planner supports at least everything of F

[jira] [Created] (FLINK-12648) Load Hadoop file system via FileSystem.get()

2019-05-28 Thread Victor Wong (JIRA)
Victor Wong created FLINK-12648: --- Summary: Load Hadoop file system via FileSystem.get() Key: FLINK-12648 URL: https://issues.apache.org/jira/browse/FLINK-12648 Project: Flink Issue Type: Improv

[jira] [Created] (FLINK-12647) Add feature flag to revert to previous behavior

2019-05-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12647: Summary: Add feature flag to revert to previous behavior Key: FLINK-12647 URL: https://issues.apache.org/jira/browse/FLINK-12647 Project: Flink Issue

Re: [DISCUSS] FLIP-33: Standardize connector metrics

2019-05-28 Thread Piotr Nowojski
Hi, Thanks for the proposal and driving the effort here Becket :) I’ve read through the FLIP-33 [1], and here are couple of my thoughts. Big +1 for standardising the metric names between connectors, it will definitely help us and users a lot. Issues/questions/things to discuss that I’ve though

[jira] [Created] (FLINK-12646) Fix broken tests of RestClientTest

2019-05-28 Thread Victor Wong (JIRA)
Victor Wong created FLINK-12646: --- Summary: Fix broken tests of RestClientTest Key: FLINK-12646 URL: https://issues.apache.org/jira/browse/FLINK-12646 Project: Flink Issue Type: Bug Co

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-28 Thread Till Rohrmann
+1 (binding) * verified checksum and signature Cheers, Till On Mon, May 27, 2019 at 4:24 PM Hequn Cheng wrote: > +1 (non-binding) > > - Release notes are correct. > - Built from source archive successfully. > - Signatures and hash are correct. > - All artifacts(11 artifacts including flink-sha

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread Tzu-Li (Gordon) Tai
Hi Jincheng, Thanks for starting the discussion. +1 to have a 1.8.1, given there are some critical bugs fixed already (including regressions such as FLINK-12301). I've also added https://issues.apache.org/jira/browse/FLINK-11107 as a potential blocker for 1.8.1. Will make sure this gets merged f

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread zhijiang
Thanks for launching this discussion Jincheng. +1 for releasing 1.8.1 and wish the following deadlock bug [1] reported by users would be coverd in this release. I could submit the fix today. [1] https://issues.apache.org/jira/browse/FLINK-12544 Best, Zhijiang ---

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread Till Rohrmann
Thanks for starting this discussion Jincheng. The 1.8.0 release has happened almost 2 months ago and we have already 24 fixes for 1.8.1. Thus, I think it's time for the first bug fix release for 1.8. I will review FLINK-11059 and make sure that it will go into 1.8.1 Cheers, Till On Tue, May 28,

Re: Distributed cache fault

2019-05-28 Thread Till Rohrmann
Hi Vasyl, please post these kind of question to Flink's user ML since the dev ML is used for development discussions. For the failure on Windows could you share the complete stack trace to see where exactly it fails? It looks as if on Windows the scheme part of the URI makes problems. Looking at

Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml

2019-05-28 Thread Till Rohrmann
+1 for removing it. I think it is effectively dead by now. Cheers, Till On Mon, May 27, 2019 at 4:00 PM Hequn Cheng wrote: > Hi Shaoxuan, > > Thanks a lot for driving this. +1 to remove the module. > > The git log of this module shows that it has been inactive for a long > time. I think it's ok

[jira] [Created] (FLINK-12645) high parallelism e2e test fails due to missing disk space

2019-05-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12645: Summary: high parallelism e2e test fails due to missing disk space Key: FLINK-12645 URL: https://issues.apache.org/jira/browse/FLINK-12645 Project: Flink

[jira] [Created] (FLINK-12643) Implement ExecutionGraph to FailoverTopology Adapter

2019-05-28 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12643: --- Summary: Implement ExecutionGraph to FailoverTopology Adapter Key: FLINK-12643 URL: https://issues.apache.org/jira/browse/FLINK-12643 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-12644) Setup Java 9 cron jobs

2019-05-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12644: Summary: Setup Java 9 cron jobs Key: FLINK-12644 URL: https://issues.apache.org/jira/browse/FLINK-12644 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12642) OutputBufferPoolUsageGauge can fail with NPE

2019-05-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12642: Summary: OutputBufferPoolUsageGauge can fail with NPE Key: FLINK-12642 URL: https://issues.apache.org/jira/browse/FLINK-12642 Project: Flink Issue Ty

[jira] [Created] (FLINK-12641) Add partition release logic from JM -> TM

2019-05-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12641: Summary: Add partition release logic from JM -> TM Key: FLINK-12641 URL: https://issues.apache.org/jira/browse/FLINK-12641 Project: Flink Issue Type:

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread jincheng sun
Thanks for your contribution @Aitozi, I am glad to help you push this forward ASAP. :) Aitozi 于2019年5月28日周二 下午3:20写道: > Hi, jincheng > > Thanks for bringing up this discussion, I'am working on this blocker issue > ([FLINK-12297] Clean the closure for OutputTags in PatternStream), but it > has no

[jira] [Created] (FLINK-12640) mvn install flink-shaded-hadoop2-uber: Error creating shaded jar: null

2019-05-28 Thread xiezhiqiang (JIRA)
xiezhiqiang created FLINK-12640: --- Summary: mvn install flink-shaded-hadoop2-uber: Error creating shaded jar: null Key: FLINK-12640 URL: https://issues.apache.org/jira/browse/FLINK-12640 Project: Flink

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread Aitozi
Hi, jincheng Thanks for bringing up this discussion, I'am working on this blocker issue ([FLINK-12297] Clean the closure for OutputTags in PatternStream), but it has not been merged due to lack of reviewers, I think it should be solved in 1.8.1. Thanks, Aitozi -- Sent from: http://apache-flin

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-28 Thread jincheng sun
As the same with before, I also need help to officially publishing this release. I appreciate it if someone can help me at the final stage of the 1.8.1 release. :) jincheng sun 于2019年5月28日周二 下午2:30写道: > Hi Flink devs, > > What do you think about releasing Flink 1.8.1 soon? > > We already have so

Re: [DISCUSS] FLIP to Restructure, Update & Rework Apache Flink's Documentation

2019-05-28 Thread Konstantin Knauf
Hi everyone, thank you all for your comments. I agree with Fabian, there are a lot of independent contributions subsumed in this FLIP (like Concepts or Getting Started), which indepdently will already make a big difference for new users. It looks like overall, the FLIP is accepted. Therefore, I h

[jira] [Created] (FLINK-12639) FLIP-42: Rework Documentation

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12639: Summary: FLIP-42: Rework Documentation Key: FLINK-12639 URL: https://issues.apache.org/jira/browse/FLINK-12639 Project: Flink Issue Type: Improvement