Apply for contributor permission

2019-05-29 Thread Piyush Goyal
Hi, Could someone add me as a contributor? My JIRA username is pgoyal. Thanks!

[RESULT] [VOTE] Release flink-shaded 7.0, release candidate #3

2019-05-29 Thread jincheng sun
Hi all, I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 3 of which are binding: * Chesnay * Timo * Hequn * Till * Nico * Jincheng There are no disapproving votes. Thanks everyone! Cheers, Jincheng

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-29 Thread jincheng sun
Hi, All Thanks to everyone for checking the release and voting! I’ll tally the results in a separate email. :) Best, Jincheng jincheng sun 于2019年5月29日周三 下午4:52写道: > > +1 (non-binding) > > @Nico, Great thanks for your detail check! > @Chesnay Great thanks for your explanation! > > I also do so

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread Paul Lam
Hi Seth, Sorry for the confusion. I mean currently we need to know the operator id, state name and the state type (eg. ListState, MapState) beforehand to get the states. Is possible that we can perform a scan to get all existing operator ids or state names in the savepoint? It would be good to

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread Seth Wiesman
Hi Paul, I’m not following, could you provide and example of the kind of operation your describing? Seth > On May 29, 2019, at 7:37 PM, Paul Lam wrote: > > Hi Seth, > > +1 from my side. > > I was wondering if we can add a reader method to provide a full view of the > states instead of t

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread Terry Wang
Hi Seth, Big +1 from my side. I like this idea. IMO, it’s better to chose another flip name instead of ‘connector’, which is a little confusing. > 在 2019年5月30日,上午10:37,Paul Lam 写道: > > Hi Seth, > > +1 from my side. > > I was wondering if we can add a reader method to provide a full view of t

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread Paul Lam
Hi Seth, +1 from my side. I was wondering if we can add a reader method to provide a full view of the states instead of the state of a specific operator? It would be helpful when there is some unrestored states of a previously removed operator in the savepoint. Best, Paul Lam > 在 2019年5月30日

[jira] [Created] (FLINK-12681) Make SimpleCounter thread-safe.

2019-05-29 Thread Piyush Goyal (JIRA)
Piyush Goyal created FLINK-12681: Summary: Make SimpleCounter thread-safe. Key: FLINK-12681 URL: https://issues.apache.org/jira/browse/FLINK-12681 Project: Flink Issue Type: Improvement

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread vino yang
Hi Seth, Glad to see this FLIP, big +1 for this feature! Best, Vino Seth Wiesman 于2019年5月30日周四 上午7:14写道: > Hey Everyone! > ​ > Gordon and I have been discussing adding a savepoint connector to flink > for reading, writing and modifying savepoints. > ​ > This is useful for: > ​ > Analyzing

[jira] [Created] (FLINK-12680) Add timer type to metric system

2019-05-29 Thread Piyush Goyal (JIRA)
Piyush Goyal created FLINK-12680: Summary: Add timer type to metric system Key: FLINK-12680 URL: https://issues.apache.org/jira/browse/FLINK-12680 Project: Flink Issue Type: Improvement

[Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread Seth Wiesman
Hey Everyone! ​ Gordon and I have been discussing adding a savepoint connector to flink for reading, writing and modifying savepoints. ​ This is useful for: ​ Analyzing state for interesting patterns Troubleshooting or auditing jobs by checking for discrepancies in state Bootstrapping

[Discuss] FLIP-43: Savepoint Connector

2019-05-29 Thread Seth Wiesman
Hey Everyone! Gordon and I have been discussing adding a savepoint connector to flink for reading, writing, and modifying savepoints. This is useful for: * Analyzing state for interesting patterns * Troubleshooting or auditing jobs by checking for discrepancies in state * Bootstrapping state fo

[DISCUSS] FLIP-42: Savepoint Connector

2019-05-29 Thread Seth Wiesman
Hey Everyone! Gordon and I have been discussing adding a savepoint connector to flink for reading, writing and modifying savepoints. This is useful for: - Analyzing state for interesting patterns - Troubleshooting or auditing jobs by checking for discrepancies in state - Bootstrapping s

[jira] [Created] (FLINK-12679) Support 'default-database' config for catalog entries in SQL CLI yaml file

2019-05-29 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12679: Summary: Support 'default-database' config for catalog entries in SQL CLI yaml file Key: FLINK-12679 URL: https://issues.apache.org/jira/browse/FLINK-12679 Project: Flink

[jira] [Created] (FLINK-12678) Add AbstractCatalog to manage the common catalog name and default database name for catalogs

2019-05-29 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12678: Summary: Add AbstractCatalog to manage the common catalog name and default database name for catalogs Key: FLINK-12678 URL: https://issues.apache.org/jira/browse/FLINK-12678

[jira] [Created] (FLINK-12677) Add descriptor, validator, and factory for HiveCatalog

2019-05-29 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12677: Summary: Add descriptor, validator, and factory for HiveCatalog Key: FLINK-12677 URL: https://issues.apache.org/jira/browse/FLINK-12677 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-12676) add descriptor, validator, and factory for GenericInMemoryCatalog

2019-05-29 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12676: Summary: add descriptor, validator, and factory for GenericInMemoryCatalog Key: FLINK-12676 URL: https://issues.apache.org/jira/browse/FLINK-12676 Project: Flink Is

[jira] [Created] (FLINK-12675) Event time synchronization in Kafka consumer

2019-05-29 Thread Thomas Weise (JIRA)
Thomas Weise created FLINK-12675: Summary: Event time synchronization in Kafka consumer Key: FLINK-12675 URL: https://issues.apache.org/jira/browse/FLINK-12675 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-12674) BoundedBlockingSubpartition#unsynchronizedGetNumberOfQueuedBuffers is not implemented

2019-05-29 Thread Piotr Nowojski (JIRA)
Piotr Nowojski created FLINK-12674: -- Summary: BoundedBlockingSubpartition#unsynchronizedGetNumberOfQueuedBuffers is not implemented Key: FLINK-12674 URL: https://issues.apache.org/jira/browse/FLINK-12674

[jira] [Created] (FLINK-12673) Introduce NetworkEnvironment.getUnreleasedPartitions instead of using getResultPartitionManager

2019-05-29 Thread Andrey Zagrebin (JIRA)
Andrey Zagrebin created FLINK-12673: --- Summary: Introduce NetworkEnvironment.getUnreleasedPartitions instead of using getResultPartitionManager Key: FLINK-12673 URL: https://issues.apache.org/jira/browse/FLINK-12

[jira] [Created] (FLINK-12672) Use SingleInputGate.closeFuture in triggerPartitionStateCheck instead of consumer state in RemoteChannelStateChecker

2019-05-29 Thread Andrey Zagrebin (JIRA)
Andrey Zagrebin created FLINK-12672: --- Summary: Use SingleInputGate.closeFuture in triggerPartitionStateCheck instead of consumer state in RemoteChannelStateChecker Key: FLINK-12672 URL: https://issues.apache.or

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-29 Thread Hequn Cheng
Hi Jincheng, Thanks for putting these together with a nice document. +1 to release 1.8.1. I think it would be nice if we can have a new release with so many fixes. Best, Hequn On Wed, May 29, 2019 at 5:25 PM jincheng sun wrote: > Hi all, > Thank you for your support of the release of 1.8.1. >

Key state does not support migration

2019-05-29 Thread Richard Deurwaarder
Hello, I am running into the problem where (avro) schema evolution works perfectly for operator/keyed state but does not work when used with keyBy(). For example: I have a job like so: env.addSource(someSource()) .keyBy(object -> getMyAvroObject()) .process(doSomething()

[jira] [Created] (FLINK-12671) Summarizer: summary statistics for Table

2019-05-29 Thread Xu Yang (JIRA)
Xu Yang created FLINK-12671: --- Summary: Summarizer: summary statistics for Table Key: FLINK-12671 URL: https://issues.apache.org/jira/browse/FLINK-12671 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-29 Thread jincheng sun
Hi all, Thank you for your support of the release of 1.8.1. @Till Rohrmann Thank you very much for your help review FLINK-11059! @Zhijiang Thank you feedback the very important bug fix. I'll add it to the trace list! @Tzu-Li (Gordon) Tai Great thanks for your can kindly help for the final stage

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-29 Thread Yuzhao Chen
+1, thanks for you nice work, Yu Li ! Best, Danny Chan 在 2019年5月24日 +0800 PM8:51,Yu Li ,写道: > Hi All, > > As mentioned in our speak[1] given in FlinkForwardChina2018, we have > improved HeapKeyedStateBackend to support disk spilling and put it in > production here in Alibaba for last year's Single

[jira] [Created] (FLINK-12670) Implement FailureRateRestartBackoffTimeStrategy

2019-05-29 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-12670: - Summary: Implement FailureRateRestartBackoffTimeStrategy Key: FLINK-12670 URL: https://issues.apache.org/jira/browse/FLINK-12670 Project: Flink Issue Type:

[jira] [Created] (FLINK-12669) Implement FixedDelayRestartBackoffTimeStrategy

2019-05-29 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-12669: - Summary: Implement FixedDelayRestartBackoffTimeStrategy Key: FLINK-12669 URL: https://issues.apache.org/jira/browse/FLINK-12669 Project: Flink Issue Type:

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-29 Thread jincheng sun
+1 (non-binding) @Nico, Great thanks for your detail check! @Chesnay Great thanks for your explanation! I also do some check regarding the change which Nico mentioned above, share my result here: - Jackson 2.7.9 upgrade to 2.9.8 brings some changes, including fewer classes in `snakeanaml`. This

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-29 Thread Stefan Richter
Hi Yu, Sorry for the late reaction. As already discussed internally, I think this is a very good proposal and design that can help to improve a major limitation of the current state backend. I think that most discussion is happening in the design doc and I left my comments there. Looking forwar

Re: [DISCUSS] Flink client api enhancement for downstream project

2019-05-29 Thread Jeff Zhang
Sorry folks, the design doc is late as you expected. Here's the design doc I drafted, welcome any comments and feedback. https://docs.google.com/document/d/1VavBrYn8vJeZs-Mhu5VzKO6xrWCF40aY0nlQ_UVVTRg/edit?usp=sharing Stephan Ewen 于2019年2月14日周四 下午8:43写道: > Nice that this discussion is happeni

[jira] [Created] (FLINK-12668) Introduce fromParallelElements for generating DataStreamSource

2019-05-29 Thread bupt_ljy (JIRA)
bupt_ljy created FLINK-12668: Summary: Introduce fromParallelElements for generating DataStreamSource Key: FLINK-12668 URL: https://issues.apache.org/jira/browse/FLINK-12668 Project: Flink Issue

[jira] [Created] (FLINK-12667) Add JobID argument to TaskExecutorGateway#releasePartitions

2019-05-29 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12667: Summary: Add JobID argument to TaskExecutorGateway#releasePartitions Key: FLINK-12667 URL: https://issues.apache.org/jira/browse/FLINK-12667 Project: Flink

[jira] [Created] (FLINK-12666) Update RM idleness check for TMs to take partitions into account

2019-05-29 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12666: Summary: Update RM idleness check for TMs to take partitions into account Key: FLINK-12666 URL: https://issues.apache.org/jira/browse/FLINK-12666 Project: Fli