[jira] [Created] (FLINK-10457) Support SequenceFile for StreamingFileSink

2018-09-28 Thread Jihyun Cho (JIRA)
Jihyun Cho created FLINK-10457: -- Summary: Support SequenceFile for StreamingFileSink Key: FLINK-10457 URL: https://issues.apache.org/jira/browse/FLINK-10457 Project: Flink Issue Type: New Featur

Re: Handling burst I/O when using tumbling/sliding windows

2018-09-28 Thread Piotr Nowojski
Hi, Re 1. Can you be more specific? What system are you using, what’s happening and how does it brake? While delaying windows firing is probably the most cost effective solution for this particular problem, it has some disadvantages: a) putting even more logic to already complicated component b

Re: Discuss about [FLINK-10134] UTF-16 support for TextInputFormat bug fixed

2018-09-28 Thread Chesnay Schepler
Please move this discussion to either the PR. There's little value in spreading discussions over several channels; any insight raised here should also be visible in the PR. On 28.09.2018 07:07, x1q1j1 wrote: dear everyone, I hope to discuss this jira with everyone in order to ha

[jira] [Created] (FLINK-10458) Enable dependencyConvergence for flink-shaded

2018-09-28 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10458: Summary: Enable dependencyConvergence for flink-shaded Key: FLINK-10458 URL: https://issues.apache.org/jira/browse/FLINK-10458 Project: Flink Issue T

Re: Enrich testing doc with more unit test examples using AbstractStreamOperator

2018-09-28 Thread Tony Wei
Hi all, @Ken, Thanks for your positive feedback. I have a similar experience with test harness and that's why I want to provide more contents on testing doc to prevent from this kind of problems. Does anyone have any feedback and advice? I would like to collect more opinions from every developers

Re: Handling burst I/O when using tumbling/sliding windows

2018-09-28 Thread Rong Rong
Hi Piotrek, Thanks for getting back to me so quickly. Let me explain. Re 1). As I explained in the doc. we are using a basic Kafka-in Kafka-out system with same partition number on both side. It is causing degraded performance in external I/O network traffic. It is definitely possible to configur

[DISCUSS] Dropping flink-storm?

2018-09-28 Thread Till Rohrmann
Hi everyone, I would like to discuss how to proceed with Flink's storm compatibility layer flink-strom. While working on removing Flink's legacy mode, I noticed that some parts of flink-storm rely on the legacy Flink client. In fact, at the moment flink-storm does not work together with Flink's n

Re: Enrich testing doc with more unit test examples using AbstractStreamOperator

2018-09-28 Thread Till Rohrmann
Hi Tony, I think this is a long sought-after feature to provide better testing tools for our users. Thus, I'm strongly in favour of adding something like this. If I remember correctly Aljoscha already spend some brain cycles on this and he also gave a training about the current state at FF SF 2018

[DISCUSS] Feature freeze for Flink 1.7

2018-09-28 Thread Till Rohrmann
Dear community, almost 2 months have passed since the Flink 1.6 release and some good features for the next major release are close to completion. Therefore, I would like to discuss a possible feature freeze for Flink 1.7. Given that we've released Flink 1.6 at the beginning of August, I would li

Re: [DISCUSS] Dropping flink-storm?

2018-09-28 Thread Chesnay Schepler
I'm very much in favor of dropping it. Flink has been continually growing in terms of features, and IMO we've reached the point where we should cull some of the more obscure ones. flink-storm, while interesting from a theoretical standpoint, offers too little value. Note that the bolt/spout

Re: Handling burst I/O when using tumbling/sliding windows

2018-09-28 Thread Piotr Nowojski
Hi, Thanks for the response again :) Re 1). Do you mean that this extra burst external I/O network traffic is causing disturbance with other systems reading/writing from Kafka? With Kafka itself? Re 2a) Yes, it should be relatively simple, however any new brick makes the overall component mo

Re: Enrich testing doc with more unit test examples using AbstractStreamOperator

2018-09-28 Thread Tony Wei
Hi Till, Thanks for your feedback. I didn't think about this before. It will be better if we can provide such tools instead of let user deal with testing in operator level directly. Because test harness was introduced for contributors, who may have more knowledge about internal design, to test the

[jira] [Created] (FLINK-10459) Flink cluster does not work properly in kubernetes

2018-09-28 Thread Georgii Gobozov (JIRA)
Georgii Gobozov created FLINK-10459: --- Summary: Flink cluster does not work properly in kubernetes Key: FLINK-10459 URL: https://issues.apache.org/jira/browse/FLINK-10459 Project: Flink Issu

[jira] [Created] (FLINK-10460) DataDog reporter JsonMappingException

2018-09-28 Thread Elias Levy (JIRA)
Elias Levy created FLINK-10460: -- Summary: DataDog reporter JsonMappingException Key: FLINK-10460 URL: https://issues.apache.org/jira/browse/FLINK-10460 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Dropping flink-storm?

2018-09-28 Thread vino yang
Hi, +1, I agree. In addition, some users ask questions about the integration of Storm compatibility mode with the newer Flink version on the mailing list. It seems that they are not aware that some of Flink's new features are no longer available in Storm compatibility mode. This can be confusing

Re: [DISCUSS] Dropping flink-storm?

2018-09-28 Thread Hequn Cheng
Hi, +1 to drop it. It seems that few people use it. Best, Hequn On Fri, Sep 28, 2018 at 10:22 PM Chesnay Schepler wrote: > I'm very much in favor of dropping it. > > Flink has been continually growing in terms of features, and IMO we've > reached the point where we should cull some of the more

Re: [DISCUSS] Dropping flink-storm?

2018-09-28 Thread Jeff Carter
+1 to drop it. On Fri, Sep 28, 2018, 7:25 PM Hequn Cheng wrote: > Hi, > > +1 to drop it. It seems that few people use it. > > Best, Hequn > > On Fri, Sep 28, 2018 at 10:22 PM Chesnay Schepler > wrote: > > > I'm very much in favor of dropping it. > > > > Flink has been continually growing in ter

[jira] [Created] (FLINK-10461) Speed up download file procedure when restore

2018-09-28 Thread Congxian Qiu (JIRA)
Congxian Qiu created FLINK-10461: Summary: Speed up download file procedure when restore Key: FLINK-10461 URL: https://issues.apache.org/jira/browse/FLINK-10461 Project: Flink Issue Type: Im

Re: [DISCUSS] Feature freeze for Flink 1.7

2018-09-28 Thread Steven Wu
Please prioritize a proper long-term fix for this issue. it is a big scalability issue for high-parallelism job (e.g. over 1,000). FLINK-10122 KafkaConsumer should use partitionable state over union state if partition discovery is not active On Fri, Sep 28, 2018 at 7:20 AM Till Rohrmann wrote:

[jira] [Created] (FLINK-10462) Remove ConnectionIndex for further sharing tcp connection in credit-based mode

2018-09-28 Thread zhijiang (JIRA)
zhijiang created FLINK-10462: Summary: Remove ConnectionIndex for further sharing tcp connection in credit-based mode Key: FLINK-10462 URL: https://issues.apache.org/jira/browse/FLINK-10462 Project: Flin

回复:[DISCUSS] Dropping flink-storm?

2018-09-28 Thread Zhijiang(wangzhijiang999)
Very agree with to drop it. +1 -- 发件人:Jeff Carter 发送时间:2018年9月29日(星期六) 10:18 收件人:dev 抄 送:chesnay ; Till Rohrmann ; user 主 题:Re: [DISCUSS] Dropping flink-storm? +1 to drop it. On Fri, Sep 28, 2018, 7:25 PM Hequn Cheng wrote: > H

Re: [DISCUSS] Dropping flink-storm?

2018-09-28 Thread 远远
+1, it‘s time to drop it😂 Zhijiang(wangzhijiang999) 于2018年9月29日周六 下午1:53写道: > Very agree with to drop it. +1 > > -- > 发件人:Jeff Carter > 发送时间:2018年9月29日(星期六) 10:18 > 收件人:dev > 抄 送:chesnay ; Till Rohrmann ; > user > 主 题:Re: [DISCUS

Re: [DISCUSS] Dropping flink-storm?

2018-09-28 Thread Tzu-Li Chen
+1 to drop it. It seems few people use it. Commits history of an experimental module sparse often means that there is low interest. Best, tison. 远远 于2018年9月29日周六 下午2:16写道: > +1, it‘s time to drop it😂 > > Zhijiang(wangzhijiang999) 于2018年9月29日周六 > 下午1:53写道: > >> Very agree with to drop it. +1