Hi Yangze, Thanks for the clarification. The example of two batch jobs team up with one streaming job is interesting.
Best regards, Jing On Wed, Sep 20, 2023 at 7:19 PM Yangze Guo <karma...@gmail.com> wrote: > Thanks for the comments, Jing. > > > Will the minimum resource configuration also take effect for streaming > jobs in application mode? > > Since it is not recommended to configure slotmanager.number-of-slots.max > for streaming jobs, does it make sense to disable it for common streaming > jobs? At least disable the check for avoiding the oscillation? > > Yes. The minimum resource configuration will only disabled in > standalone cluster atm. I agree it make sense to disable it for a pure > streaming job, however: > - By default, the minimum resource is configured to 0. If users do not > proactively set it, either the oscillation check or the minimum > restriction can be considered as disabled. > - The minimum resource is a cluster-level configuration rather than a > job-level configuration. If a user has an application with two batch > jobs preceding the streaming job, they may also require this > configuration to accelerate the execution of batch jobs. > > WDYT? > > Best, > Yangze Guo > > On Thu, Sep 21, 2023 at 4:49 AM Jing Ge <j...@ververica.com.invalid> > wrote: > > > > Hi Xiangyu, > > > > Thanks for driving it! There is one thing I am not really sure if I > > understand you correctly. > > > > According to the FLIP: "The minimum resource limitation will be > implemented > > in the DefaultResourceAllocationStrategy of FineGrainedSlotManager. > > > > Each time when SlotManager needs to reconcile the cluster resources or > > fulfill job resource requirements, the DefaultResourceAllocationStrategy > > will check if the minimum resource requirement has been fulfilled. If it > is > > not, DefaultResourceAllocationStrategy will request new > PendingTaskManagers > > and FineGrainedSlotManager will allocate new worker resources > accordingly." > > > > "To avoid this oscillation, we need to check the worker number derived > from > > minimum and maximum resource configuration is consistent before starting > > SlotManager." > > > > Will the minimum resource configuration also take effect for streaming > jobs > > in application mode? Since it is not recommended to > > configure slotmanager.number-of-slots.max for streaming jobs, does it > make > > sense to disable it for common streaming jobs? At least disable the check > > for avoiding the oscillation? > > > > Best regards, > > Jing > > > > > > On Tue, Sep 19, 2023 at 4:58 PM Chen Zhanghao <zhanghao.c...@outlook.com > > > > wrote: > > > > > Thanks for driving this, Xiangyu. We use Session clusters for quick SQL > > > debugging internally, and found cold-start job submission slow due to > lack > > > of the exact minimum resource reservation feature proposed here. This > > > should improve the experience a lot for running short lived-jobs in > session > > > clusters. > > > > > > Best, > > > Zhanghao Chen > > > ________________________________ > > > 发件人: Yangze Guo <karma...@gmail.com> > > > 发送时间: 2023年9月19日 13:10 > > > 收件人: xiangyu feng <xiangyu...@gmail.com> > > > 抄送: dev@flink.apache.org <dev@flink.apache.org> > > > 主题: Re: [Discuss] FLIP-362: Support minimum resource limitation > > > > > > Thanks for driving this @Xiangyu. This is a feature that many users > > > have requested for a long time. +1 for the overall proposal. > > > > > > Best, > > > Yangze Guo > > > > > > On Tue, Sep 19, 2023 at 11:48 AM xiangyu feng <xiangyu...@gmail.com> > > > wrote: > > > > > > > > Hi Devs, > > > > > > > > I'm opening this thread to discuss FLIP-362: Support minimum resource > > > limitation. The design doc can be found at: > > > > FLIP-362: Support minimum resource limitation > > > > > > > > Currently, the Flink cluster only requests Task Managers (TMs) when > > > there is a resource requirement, and idle TMs are released after a > certain > > > period of time. However, in certain scenarios, such as running short > > > lived-jobs in session cluster and scheduling batch jobs stage by > stage, we > > > need to improve the efficiency of job execution by maintaining a > certain > > > number of available workers in the cluster all the time. > > > > > > > > After discussed with Yangze, we introduced this new feature. The new > > > added public options and proposed changes are described in this FLIP. > > > > > > > > Looking forward to your feedback, thanks. > > > > > > > > Best regards, > > > > Xiangyu > > > > > > > >
