回复：[DISCUSS] Improve broadcast serialization

-- 发件人：Piotr Nowojski 发送时间：2018年10月18日(星期四) 17:47 收件人：Zhijiang(wangzhijiang999) 抄　送：Nico Kruber ; dev 主　题：Re: [DISCUSS] Improve broadcast serialization Hey, I also think that 3rd option is the most promising, however logic of “dirty” channels might be

回复：Sharing state between subtasks

(wangzhijiang999) 抄　送：Till Rohrmann 主　题：Re: Sharing state between subtasks Hi Zhijiang, do you already have working code or a design doc for the second approach? Best, Aljoscha > On 18. Oct 2018, at 08:42, Zhijiang(wangzhijiang999) > wrote: > > Just noticed this discussion from @Till Rohrm

回复：Sharing state between subtasks

Just noticed this discussion from @Till Rohrmann's weekly community update and I want to share some thoughts from our experiences. We also encountered the source consuption skew issue before, and we are focused on improving this by two possible ways. 1. Control the read strategy by the downstre

回复：[DISCUSS] Improve broadcast serialization

nt thoughting, which keeps the same behavior like normal "emit" mixing with "broadcastEvent". Best, Zhijiang -- 发件人：Piotr Nowojski 发送时间：2018年10月17日(星期三) 19:25 收件人：Zhijiang(wangzhijiang999) 抄　送：Nico Kruber ; dev 主　题：Re: [DISCUSS] Improve broadc

回复：[DISCUSS] Improve broadcast serialization

Best, Zhijiang -- 发件人：Zhijiang(wangzhijiang999) 发送时间：2018年7月20日(星期五) 13:21 收件人：Piotr Nowojski 抄　送：Nico Kruber ; dev 主　题：回复：[DISCUSS] Improve broadcast serialization Ok, that is fine. :) I will create JIRA today and submit the PR

回复：[DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Thanks Fabian for proposing this topic. It is very worth improving the web dashborad for showing more useful informations which can benefit flink users a lot. Just two small personal concerns: 1. The start time and end time are already given, so it is easy to estimate the rough duration time. I

回复：[DISCUSS] [Contributing] (2) - Review Steps

+1 -- 发件人：vino yang 发送时间：2018年10月9日(星期二) 14:08 收件人：dev 主　题：Re: [DISCUSS] [Contributing] (2) - Review Steps +1 Peter Huang 于2018年10月9日周二 下午1:54写道： > +1 > > On Mon, Oct 8, 2018 at 7:47 PM Thomas Weise wrote: > > > +1 > > > > > > O

回复：[DISCUSS] Dropping flink-storm?

Very agree with to drop it. +1 -- 发件人：Jeff Carter 发送时间：2018年9月29日(星期六) 10:18 收件人：dev 抄　送：chesnay ; Till Rohrmann ; user 主　题：Re: [DISCUSS] Dropping flink-storm? +1 to drop it. On Fri, Sep 28, 2018, 7:25 PM Hequn Cheng wrote: > H

回复：Codespeed deployment for Flink

Thanks @Piotr Nowojski and @Nico Kruber for the good job! I already benefit from this benchmark in the previous PRs. Wish the visualization tool becoming stronger to benefit more for the community! Best, Zhijiang -- 发件人：Piotr Nowoj

回复：[PROPOSAL] [community] A more structured approach to reviews and contributions

From my personal experience as a contributor for three years, I feel better experience in contirbuting or reviewing than before, although we still have some points for further progress. I reviewed the proposal doc, and it gives very constructive and meaningful guides which could help both contr

回复：[DISCUSS] Proposal of external shuffle service

Many thanks Till! I would create a JIRA for this feature and design a document attched with it. I will let you know after ready! :) Best, Zhijiang -- 发件人：Till Rohrmann 发送时间：2018年9月7日(星期五) 22:01 收件人：Zhijiang(wangzhijiang999) 抄　送

回复：[ANNOUNCE] New committer Gary Yao

Congratulations Gary! :) -- 发件人：Piotr Nowojski 发送时间：2018年9月10日(星期一) 16:31 收件人：dev 主　题：Re: [ANNOUNCE] New committer Gary Yao Congratulations :) > On 8 Sep 2018, at 07:03, Rong Rong wrote: > > Congratulations Gary!! > > On Fri, S

回复：[DISCUSS] Proposal of external shuffle service

间：2018年8月29日(星期三) 17:36 收件人：dev ; Zhijiang(wangzhijiang999) 主　题：Re: [DISCUSS] Proposal of external shuffle service Thanks for starting this design discussion Zhijiang! I really like the idea to introduce a ShuffleService abstraction which allows to have different implementations depending on

[DISCUSS] Proposal of external shuffle service

Hi all! The shuffle service is responsible for transporting upstream produced data to the downstream side. In flink, the NettyServer is used for network transport service and this component is started in the TaskManager process. That means the TaskManager can support internal shuffle service wh

回复：[DISCUSS] Improve broadcast serialization

Ok, that is fine. :) I will create JIRA today and submit the PR next week. Zhijiang -- 发件人：Piotr Nowojski 发送时间：2018年7月19日(星期四) 17:52 收件人：Zhijiang(wangzhijiang999) 抄　送：Nico Kruber ; dev 主　题：Re: [DISCUSS] Improve broadcast

回复：[DISCUSS] Improve broadcast serialization

18日(星期三) 20:04 收件人：Zhijiang(wangzhijiang999) ; Nico Kruber 抄　送：dev 主　题：Re: [DISCUSS] Improve broadcast serialization Hi 1. I want to define a new AbstractRecordWriter as base class which defines some abstract methods and utility codes. The current RecordWriter used for other partitioner and

回复：[DISCUSS] Improve broadcast serialization

think of other ways to improve copy or confirm finish non-full BufferBuilder no harms. I think we can get good benefits from serialization cost and memory overhead currently. Best, Zhijiang -- 发件人：Zhijiang(wangzhijiang999) 发送时间

回复：[DISCUSS] Improve broadcast serialization

reate jiras after we reach a final agreement, then maybe you can help review PR if have time. :) Best, Zhijiang -- 发件人：Piotr Nowojski 发送时间：2018年7月18日(星期三) 16:37 收件人：dev ; Zhijiang(wangzhijiang999) 主　题：Re: [DISCUSS] Improve broa

回复：[DISCUSS] Improve broadcast serialization

发送时间：2018年7月17日(星期二) 23:31 收件人：dev ; Zhijiang(wangzhijiang999) 主　题：Re: [DISCUSS] Improve broadcast serialization Hi Generally speaking this would be a nice optimisation, however it might be tricky to implement. The thing to keep in mind is that currently interface allow to interleave broadcasti

[DISCUSS] Improve broadcast serialization

Hi all, In current implementation, the RecordSerializer is created separately for each subpartition in RecordWriter, that means the number of serializers equals to the number of subpartitions. For broadcast partitioner, every record will be serialized many times in all the subpartitions, and th

回复：[DISCUSS] Release Flink 1.5.1

I have reviewed FLINK-9676, wish it merged soon. Zhijiang -- 发件人：Chesnay Schepler 发送时间：2018年7月5日(星期四) 15:58 收件人：dev@flink.apache.org ; wangzhijiang999 ; zjffdu ; Till Rohrmann 主　题：Re: [DISCUSS] Release Flink 1.5.1 Building the bi

回复：[DISCUSS] Release Flink 1.5.1

Hi Chesnay, Agree with your proposal. I submitted a jira FLINK-9676 related with deadlock issue. I think it needs to be confirmed whether to be covered in this release or later. Zhijiang -- 发件人：Chesnay Schepler 发送时间：2018年7月2日(星期一)

回复：[ANNOUNCE] New committer Piotr Nowojski

Congrats Piotr! Feel good to cooperate with you before. :) Zhijiang -- 发件人：Ufuk Celebi 发送时间：2018年6月24日(星期日) 01:25 收件人：dev 主　题：Re: [ANNOUNCE] New committer Piotr Nowojski Congrats and welcome Piotr! :-) – Ufuk On Sat, Jun 23, 20

回复：[ANNOUNCE] Two new committers: Xingcan Cui and Nico Kruber

Congratulations, Xingcan and Nico ! Nico is a good PR reviewer and I gained a lot from him. :)--发件人：Fabian Hueske 发送时间：2018年5月9日(星期三) 02:53收件人：dev 主　题：[ANNOUNCE] Two new committers: Xingcan Cui and Nico Kruber Hi everyone, I'm hap

回复：[DISCUSS] Releasing Flink 1.4

：2017年10月18日(星期三) 22:54收件人：Till Rohrmann 抄　送：dev ; Zhijiang(wangzhijiang999) 主　题：Re: [DISCUSS] Releasing Flink 1.4 Hi Zhijiang! The proposal is to release 1.4 asap and have the master switch so 1.5 soon (early November or so). Merge the network code directly after that - the 1.5 release would come

回复：[DISCUSS] Releasing Flink 1.4

Thanks for replying @Till Rohrmann , and I agree with your suggestion. Cheers,Zhijiang --发件人：Till Rohrmann 发送时间：2017年10月18日(星期三) 16:27收件人：dev ; Zhijiang(wangzhijiang999) 抄　 送：Stephan Ewen 主　题：Re: [DISCUSS

回复：[DISCUSS] Releasing Flink 1.4

ek : >>>  >>>> @Bowen I started marking essential stuff as blocking (with fixVersion >>>> 1.4.0). You're right, that we should start moving things to 1.5.0 that >>> are >>>> not blocking and that we don't think will make it into

回复：[DISCUSS] Releasing Flink 1.4

totally agree with the way.--发件人：Stephan Ewen 发送时间：2017年10月13日(星期五) 21:29收件人：dev@flink.apache.org 主　题：Re: [DISCUSS] Releasing Flink 1.4 I am in favor of doing this, if we can set it up in the following way.   - We put out the 1.4 r

回复：[DISCUSS] A more thorough Pull Request check list and template

From my side, I also like the new template which contains more useful information, and both the reviewer and contributor may get benefits from it. For my previous pull requests as a contributor, I always listed the purpose of change and change log, but rarely mentioned how to verify the change an

回复：An addition to Netty's memory footprint

Based on Kurt's scenario, if the cumulator allocates a big ByteBuf from ByteBufAllocator during expansion, it is easy to result in creating a new PoolChunk(16M) because of no consistent memory in current PoolChunks. And this will cause the total used direct memory beyond estimated. For further

回复：[ANNOUNCE] New Flink committer Shaoxuan Wang

Congratulations, Shaoxuan! Cheers,Zhijiang--发件人：Fabian Hueske 发送时间：2017年6月22日(星期四) 04:19收件人：dev@flink.apache.org 主　题：[ANNOUNCE] New Flink committer Shaoxuan Wang Hi everybody, On behalf of the PMC, I'm very happy to announce that Sh

回复：[DISCUSS] Feature Freeze

Hi Ufuk, Thank you for launching this topic! I wish my latest refinement of buffer provider (https://issues.apache.org/jira/browse/FLINK-6337)  to be included in 1.3 and most of the jobs can get benefit from it. And I think it can be completed with the help of your reviews this week. Cheers,Zhij

回复：[jira] [Created] (FLINK-6057) Better default needed for num network buffers

Hi Luke,        Yes, it is really not very friendly for users to set the number of network buffers. And it may cause OOM exception if not set the proper amount for large scale job. The proper amount should be calculated automatically by framework based on the number of input and output channels

回复：Dataset and eager scheduling

Hi,     From my understand,  if you do not care resource waste and confirm there are enough resources in cluster, you can set EAGER schedule mode for batch job. From optimizer aspect, if not set the PIPELINED_FORCED hint for  ExecutionMode, for some special topology cases, the optimizer would