Hi Junrui, Thanks for creating this FLIP!
AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch scheduling, also with some must-have features like speculative execution. It will be great that users can easily use it, without required to knowing the underlying scheduler and configuring some advanced items. So generally +1 for this proposal. Regarding the configuration key renaming, like yuxia mentioned, we should deprecate the old ones and add new ones with new names, to guarantee compatibility. Thanks, Zhu yuxia <luoyu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道： > > Hi, JunRui Lee. > > Thanks for driving this FLIP. It must a good improvement for batch users' > experiences. > I have few questions about this FLIP: > 1: About the configuration renaming. The old configurations will be > deprecated or removed directly? if user upgrade their Flink version, these > old configuration will still be considered or just ignored? If ignore, the > users may need to modify their configurations after they upgrade their Flink. > > 2: I'm cursion in which case users will disable auto parallelism derivation > if they have enabled adaptive batch scheduler. IIUC, auto parallelism > derivation is what adaptive batch scheduler aim to do. If use want to diable > auto parallelism derivation, can they just disable adaptive batch scheduler.? > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "JunRui Lee" <jrlee....@gmail.com> > 收件人: "dev" <dev@flink.apache.org> > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for > batch jobs > > Hi, devs, > > I'd like to start a discussion about FLIP-283: Use adaptive batch > scheduler as default scheduler for batch jobs[1]. > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive > batch scheduler has stronger batch scheduling capabilities, including > automatically deciding parallelisms of job vertices for batch > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > and speculative execution (FLIP-168)[4]. To further use the adaptive > batch scheduler to improve flink's batch capability, in this FLIP > we aim to make the adaptive batch scheduler as the default batch > scheduler. > > Currently, users have to set some configuration of the adaptive > batch scheduler, which is not very convenient. To use the adaptive > batch scheduler as the default batch scheduler, we need to improve > the user's out-of-the-box experience. Therefore, we also need to > optimize the current adaptive batch scheduler configuration. > > Looking forward to your feedback. > > [1]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > [2]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > [4]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > Best regards, > JunRui Lee
