Re: [ANNOUNCE] Jincheng Sun is now part of the Flink PMC

2019-06-24 Thread Tao Yangyu
Congratulations! Jincheng well deserved! BR, Ryan Kurt Young 于2019年6月25日周二 上午10:10写道: > Congratulations Jincheng! > > Best, > Kurt > > > On Tue, Jun 25, 2019 at 9:56 AM LakeShen > wrote: > > > Congratulations! Jincheng Sun > > > > Best, > > LakeShen > > > > Robert Metzger 于2019年6月24日周一 下午11:0

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-28 Thread Tao Yangyu
Thank Bo for your interesting example, and it seems the current task retry strategy could be more efficient. The ExternalShuffleService refactor is an exciting improvement for both shuffle and failover. And Bo proposed to make the SubPartition buffers available for possible later retry until the co

Re: [DISCUSS] Task speculative execution for Flink batch

2018-11-18 Thread Tao Yangyu
ive task can be > considered as a special case of this. > > Xiaowei > > On Sat, Nov 17, 2018 at 10:27 PM Tao Yangyu wrote: > > > Hi all, > > > > After refined, the detailed design doc is here: > > > > > https://docs.google.com/document/d/1X_Pfo4WcO-TE

Re: [DISCUSS] Task speculative execution for Flink batch

2018-11-17 Thread Tao Yangyu
Hi all, After refined, the detailed design doc is here: https://docs.google.com/document/d/1X_Pfo4WcO-TEZmmVTTYNn44LQg5gnFeeaeqM7ZNLQ7M/edit?usp=sharing Your kind reviews and comments are very appreciated and will help so much the feature to be completed. Best, Ryan Tao Yangyu 于2018年11月7日周三

Re: [DISCUSS] Task speculative execution for Flink batch

2018-11-07 Thread Tao Yangyu
s proposal. We ever found this scene frequently that > some > >> long tail tasks to delay the total batch job execution time in > production. > >> We also have some thoughts for bringing this mechanism. Looking forward > to > >> your detail design doc, then we can discus

[DISCUSS] Task speculative execution for Flink batch

2018-11-05 Thread Tao Yangyu
Hi everyone, We propose task speculative execution for Flink batch in this message as follows. In the batch mode, the job is usually divided into multiple parallel tasks executed cross many nodes in the cluster. It is common to encounter the performance degradation on some nodes due to hardware p