[jira] [Created] (FLINK-12895) TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis

2019-06-18 Thread Haibo Sun (JIRA)
Haibo Sun created FLINK-12895: - Summary: TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis Key: FLINK-12895 URL: https://issues.apache.org/jira/browse/FLINK-12895

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

2019-06-18 Thread vino yang
Hi Kurt and Piotrek, Thanks for your comments. I agree that we can provide a better abstraction to be compatible with two different implementations. First of all, I think we should consider what kind of scenarios we need to support in *API* level? We have some use cases which need to a customiz

Re: About Deprecating split/select for DataStream API

2019-06-18 Thread SHI Xiaogang
Hi all, I think many Flink users are not aware of the "abnormal" behavior of current split/select implementation. As split/select are widely used, maybe we should notice our users of the abnormal behavior in the user mailing list and hear their suggestions on how to fix it. Then we can make decisi

[jira] [Created] (FLINK-12894) Document how to configure and use catalogs in SQL CLI in Chinese

2019-06-18 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12894: Summary: Document how to configure and use catalogs in SQL CLI in Chinese Key: FLINK-12894 URL: https://issues.apache.org/jira/browse/FLINK-12894 Project: Flink Iss

Apply for contributor permission

2019-06-18 Thread Jasper Yue
Hi Guys, I want to contribute to Apache Flink. Would you please give me the permission as a contributor? My JIRA ID is yuetongshu Regards, Jasper Yue

[jira] [Created] (FLINK-12893) check certain properties that are not supported when alter database in HiveCatalog

2019-06-18 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12893: Summary: check certain properties that are not supported when alter database in HiveCatalog Key: FLINK-12893 URL: https://issues.apache.org/jira/browse/FLINK-12893 Project: F

Re: About Deprecating split/select for DataStream API

2019-06-18 Thread Xingcan Cui
Hi all, @Dawid Thanks for your further explanation. I also think now we can safely move one step forward to discuss how to fix this problem, i.e., whether to (explicitly) replace the existing split/select with a new API or (implicitly) re-implement it. Note that the two options are actually n

[jira] [Created] (FLINK-12892) serialize catalog table to properties for table discovery service

2019-06-18 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12892: Summary: serialize catalog table to properties for table discovery service Key: FLINK-12892 URL: https://issues.apache.org/jira/browse/FLINK-12892 Project: Flink Is

[jira] [Created] (FLINK-12891) evaluate converting hadoop/hive writable between Hive functions and Flink

2019-06-18 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12891: Summary: evaluate converting hadoop/hive writable between Hive functions and Flink Key: FLINK-12891 URL: https://issues.apache.org/jira/browse/FLINK-12891 Project: Flink

[jira] [Created] (FLINK-12890) Add partition lifecycle related Shuffle API

2019-06-18 Thread Andrey Zagrebin (JIRA)
Andrey Zagrebin created FLINK-12890: --- Summary: Add partition lifecycle related Shuffle API Key: FLINK-12890 URL: https://issues.apache.org/jira/browse/FLINK-12890 Project: Flink Issue Type:

[jira] [Created] (FLINK-12889) Job keeps in FAILING state

2019-06-18 Thread Fan Xinpu (JIRA)
Fan Xinpu created FLINK-12889: - Summary: Job keeps in FAILING state Key: FLINK-12889 URL: https://issues.apache.org/jira/browse/FLINK-12889 Project: Flink Issue Type: Bug Components: AP

[jira] [Created] (FLINK-12888) Introduce planner rule to push filter into TableSource

2019-06-18 Thread godfrey he (JIRA)
godfrey he created FLINK-12888: -- Summary: Introduce planner rule to push filter into TableSource Key: FLINK-12888 URL: https://issues.apache.org/jira/browse/FLINK-12888 Project: Flink Issue Type

Re: Updating Kinesis Connector to latest Apache licensed libs

2019-06-18 Thread Dyana Rose
I've pushed an early WIP of the code changes to our fork and added a long comment on the work that's been done, what issues I've come across, and requests for discussion on those issues https://issues.apache.org/jira/browse/FLINK-12847 Thanks, Dyana On Fri, 14 Jun 2019 at 10:34, Aljoscha Krettek

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

2019-06-18 Thread Kurt Young
Hi Piotr, Thanks for joining the discussion. Make “local aggregation" abstract enough sounds good to me, we could implement and verify alternative solutions for use cases of local aggregation. Maybe we will find both solutions are appropriate for different scenarios. Starting from a simple one so

[jira] [Created] (FLINK-12887) Schedule UnfencedMessage would lost envelope info

2019-06-18 Thread TisonKun (JIRA)
TisonKun created FLINK-12887: Summary: Schedule UnfencedMessage would lost envelope info Key: FLINK-12887 URL: https://issues.apache.org/jira/browse/FLINK-12887 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

2019-06-18 Thread Piotr Nowojski
Hi Kurt and Vino, I think there is a trade of hat we need to consider for the local aggregation. Generally speaking I would agree with Kurt about local aggregation/pre aggregation not using Flink's state flush the operator on a checkpoint. Network IO is usually cheaper compared to Disks IO. Thi

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

2019-06-18 Thread vino yang
Hi Kurt, Thanks for your reply. If you do not depend on the window operator, that means you need to provide many Trigger related implementations like window operator. What's more, you worry about the complexity of the window operator but ignore the flexible which window operator provided for the

Re: Something wrong with travis?

2019-06-18 Thread jincheng sun
Test result: - The test for only compile state are succeeding (I deleted some old caches) cache size 1146.26M. See here https://travis-ci.org/sunjincheng121/flink/caches - timeout to 1200 test fail, get the same error, but I think maybe the storage problem, so I delete more old cache and restart t

Re: A Question About Flink Network Stack

2019-06-18 Thread bupt_ljy
Hi Zhijiang, Thank you for the detailed explaination! Best Regards, Jiayi Liao Original Message Sender:zhijiangwangzhijiang...@aliyun.com.INVALID Recipient:dev...@flink.apache.org Date:Tuesday, Jun 18, 2019 17:34 Subject:Re: A Question About Flink Network Stack Hi Jiayi, Thanks for concer

Apply for contributor permission

2019-06-18 Thread 何建宁
Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is hejianning,

Re: Something wrong with travis?

2019-06-18 Thread jincheng sun
I agree with the explanation from @Chesnay Schepler . this should be a problem with the Travis infrastructure because recently we have not big changed the logic of Travis inside Flink. At present, most of the failures are after the compile is completed. The cache size is only 7.7M, which means tha

[jira] [Created] (FLINK-12886) Support container memory segment

2019-06-18 Thread Liya Fan (JIRA)
Liya Fan created FLINK-12886: Summary: Support container memory segment Key: FLINK-12886 URL: https://issues.apache.org/jira/browse/FLINK-12886 Project: Flink Issue Type: New Feature Co

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

2019-06-18 Thread Kurt Young
Hi, For the trigger, it depends on what operator we want to use under the API. If we choose to use window operator, we should also use window's trigger. However, I also think reuse window operator for this scenario may not be the best choice. The reasons are the following: 1. As a lot of people a

[jira] [Created] (FLINK-12885) Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation

2019-06-18 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12885: Summary: Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation Key: FLINK-12885 URL: https://issues.apache.org/jira/browse/FLINK-12885 Pr

Re: A Question About Flink Network Stack

2019-06-18 Thread zhijiang
Hi Jiayi, Thanks for concerning the network stack and you pointed out a very good question. Your understanding is right. In credit-based mode, on receiver side it has fixed exclusive buffers(credits) for each remote input channel to confirm every channel could receive data in parallel, not blo

[DISCUSS] Connectors and NULL handling

2019-06-18 Thread Aljoscha Krettek
Hi All, Thanks to Gary, I recently came upon an interesting cluster of issues: - https://issues.apache.org/jira/browse/FLINK-3679: Allow Kafka consumer to skip corrupted messages - https://issues.apache.org/jira/browse/FLINK-5583: Support flexible error handling in the Kafka consumer - https:

[jira] [Created] (FLINK-12884) Implement HighAvailabilityService based on native k8s APIs

2019-06-18 Thread MalcolmSanders (JIRA)
MalcolmSanders created FLINK-12884: -- Summary: Implement HighAvailabilityService based on native k8s APIs Key: FLINK-12884 URL: https://issues.apache.org/jira/browse/FLINK-12884 Project: Flink

[jira] [Created] (FLINK-12883) Add elaborated partition release logic

2019-06-18 Thread Gary Yao (JIRA)
Gary Yao created FLINK-12883: Summary: Add elaborated partition release logic Key: FLINK-12883 URL: https://issues.apache.org/jira/browse/FLINK-12883 Project: Flink Issue Type: Sub-task

A Question About Flink Network Stack

2019-06-18 Thread bupt_ljy
Hi all, I’m not very familiar with the network part of Flink. And a question occurs to me after I read most related source codes about network. I’ve seen that Flink uses the credit-based machanism to solve the blocking problem from receivers’ side, which means that one “slow” input channel won’t

[jira] [Created] (FLINK-12882) Remove ExecutionAttemptID field from ShuffleEnvironment#createResultPartitionWriters

2019-06-18 Thread zhijiang (JIRA)
zhijiang created FLINK-12882: Summary: Remove ExecutionAttemptID field from ShuffleEnvironment#createResultPartitionWriters Key: FLINK-12882 URL: https://issues.apache.org/jira/browse/FLINK-12882 Project:

Re: Something wrong with travis?

2019-06-18 Thread Chesnay Schepler
The problem is not that bad stuff is in the cache (which is the only thing a cache cleaning solves), it is that the test stages don't download the correct one. Our compile stage uploads stuff in to the cache, and the subsequent test builds downloads it again. Whether the upload from the comp

[jira] [Created] (FLINK-12881) Add more functionalities for ML Params and ParamInfo class

2019-06-18 Thread Xu Yang (JIRA)
Xu Yang created FLINK-12881: --- Summary: Add more functionalities for ML Params and ParamInfo class Key: FLINK-12881 URL: https://issues.apache.org/jira/browse/FLINK-12881 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-44: Support Local Aggregation in Flink

2019-06-18 Thread vino yang
Hi Kurt, Thanks for your example. Now, it looks more clearly for me. >From your example code snippet, I saw the localAggregate API has three parameters: 1. key field 2. PartitionAvg 3. CountTrigger: Does this trigger comes from window package? I will compare our and your design from AP

[jira] [Created] (FLINK-12880) Support triggering checkpoint automatically when cancel a job

2019-06-18 Thread Congxian Qiu(klion26) (JIRA)
Congxian Qiu(klion26) created FLINK-12880: - Summary: Support triggering checkpoint automatically when cancel a job Key: FLINK-12880 URL: https://issues.apache.org/jira/browse/FLINK-12880 Proje

Re: Something wrong with travis?

2019-06-18 Thread Biao Liu
It has been crashed for more than 14 hours. Hope it recovers soon. Jeff Zhang 于2019年6月18日周二 下午3:21写道: > If it is travis caching issue, we can file apache infra ticket and ask them > to clean the cache. > > > > Chesnay Schepler 于2019年6月18日周二 下午3:18写道: > > > This is (hopefully a short-lived) hicc

Re: Something wrong with travis?

2019-06-18 Thread Jeff Zhang
If it is travis caching issue, we can file apache infra ticket and ask them to clean the cache. Chesnay Schepler 于2019年6月18日周二 下午3:18写道: > This is (hopefully a short-lived) hiccup on the Travis caching > infrastructure. > > There's nothing we can do to _fix_ it; if it persists we'll have to >

Re: Something wrong with travis?

2019-06-18 Thread Chesnay Schepler
This is (hopefully a short-lived) hiccup on the Travis caching infrastructure. There's nothing we can do to _fix_ it; if it persists we'll have to rework our travis setup again to not rely on caching. On 18/06/2019 08:34, Kurt Young wrote: Hi dev, I noticed that all the travis tests trigger