Thanks for drafting the FLIP and driving the discussion, Yangze. And Thanks for the feedback, Till and Chesnay.
@Till, I agree that specifying requirements for SSGs means that SSGs need to be supported in fine-grained resource management, otherwise each operator might use as many resources as the whole group. However, I cannot think of a strong reason for not supporting SSGs in fine-grained resource management. > Interestingly, if all operators have their resources properly specified, > then slot sharing is no longer needed because Flink could slice off the > appropriately sized slots for every Task individually. > So for example, if we have a job consisting of two operator op_1 and op_2 > where each op needs 100 MB of memory, we would then say that the slot > sharing group needs 200 MB of memory to run. If we have a cluster with 2 > TMs with one slot of 100 MB each, then the system cannot run this job. If > the resources were specified on an operator level, then the system could > still make the decision to deploy op_1 to TM_1 and op_2 to TM_2. Couldn't agree more that if all operators' requirements are properly specified, slot sharing should be no longer needed. I think this exactly disproves the example. If we already know op_1 and op_2 each needs 100 MB of memory, why would we put them in the same group? If they are in separate groups, with the proposed approach the system can freely deploy them to either a 200 MB TM or two 100 MB TMs. Moreover, the precondition for not needing slot sharing is having resource requirements properly specified for all operators. This is not always possible, and usually requires tremendous efforts. One of the benefits for SSG-based requirements is that it allows the user to freely decide the granularity, thus efforts they want to pay. I would consider SSG in fine-grained resource management as a group of operators that the user would like to specify the total resource for. There can be only one group in the job, 2~3 groups dividing the job into a few major parts, or as many groups as the number of tasks/operators, depending on how fine-grained the user is able to specify the resources. Having to support SSGs might be a constraint. But given that all the current scheduler implementations already support SSGs, I tend to think that as an acceptable price for the above discussed usability and flexibility. @Chesnay Will declaring them on slot sharing groups not also waste resources if the > parallelism of operators within that group are different? > Yes. It's a trade-off between usability and resource utilization. To avoid such wasting, the user can define more groups, so that each group contains less operators and the chance of having operators with different parallelism will be reduced. The price is to have more resource requirements to specify. It also seems like quite a hassle for users having to recalculate the > resource requirements if they change the slot sharing. > I'd think that it's not really workable for users that create a set of > re-usable operators which are mixed and matched in their applications; > managing the resources requirements in such a setting would be a > nightmare, and in the end would require operator-level requirements any > way. > In that sense, I'm not even sure whether it really increases usability. > - As mentioned in my reply to Till's comment, there's no reason to put multiple operators whose individual resource requirements are already known into the same group in fine-grained resource management. - Even an operator implementation is reused for multiple applications, it does not guarantee the same resource requirements. During our years of practices in Alibaba, with per-operator requirements specified for Blink's fine-grained resource management, very few users (including our specialists who are dedicated to supporting Blink users) are as experienced as to accurately predict/estimate the operator resource requirements. Most people rely on the execution-time metrics (throughput, delay, cpu load, memory usage, GC pressure, etc.) to improve the specification. To sum up: If the user is capable of providing proper resource requirements for every operator, that's definitely a good thing and we would not need to rely on the SSGs. However, that shouldn't be a *must* for the fine-grained resource management to work. For those users who are capable and do not like having to set each operator to a separate SSG, I would be ok to have both SSG-based and operator-based runtime interfaces and to only fallback to the SSG requirements when the operator requirements are not specified. However, as the first step, I think we should prioritise the use cases where users are not that experienced. Thank you~ Xintong Song On Thu, Jan 7, 2021 at 9:55 PM Chesnay Schepler <ches...@apache.org> wrote: > Will declaring them on slot sharing groups not also waste resources if > the parallelism of operators within that group are different? > > It also seems like quite a hassle for users having to recalculate the > resource requirements if they change the slot sharing. > I'd think that it's not really workable for users that create a set of > re-usable operators which are mixed and matched in their applications; > managing the resources requirements in such a setting would be a > nightmare, and in the end would require operator-level requirements any > way. > In that sense, I'm not even sure whether it really increases usability. > > My main worry is that it if we wire the runtime to work on SSGs it's > gonna be difficult to implement more fine-grained approaches, which > would not be the case if, for the runtime, they are always defined on an > operator-level. > > On 1/7/2021 2:42 PM, Till Rohrmann wrote: > > Thanks for drafting this FLIP and starting this discussion Yangze. > > > > I like that defining resource requirements on a slot sharing group makes > > the overall setup easier and improves usability of resource requirements. > > > > What I do not like about it is that it changes slot sharing groups from > > being a scheduling hint to something which needs to be supported in order > > to support fine grained resource requirements. So far, the idea of slot > > sharing groups was that it tells the system that a set of operators can > be > > deployed in the same slot. But the system still had the freedom to say > that > > it would rather place these tasks in different slots if it wanted. If we > > now specify resource requirements on a per slot sharing group, then the > > only option for a scheduler which does not support slot sharing groups is > > to say that every operator in this slot sharing group needs a slot with > the > > same resources as the whole group. > > > > So for example, if we have a job consisting of two operator op_1 and op_2 > > where each op needs 100 MB of memory, we would then say that the slot > > sharing group needs 200 MB of memory to run. If we have a cluster with 2 > > TMs with one slot of 100 MB each, then the system cannot run this job. If > > the resources were specified on an operator level, then the system could > > still make the decision to deploy op_1 to TM_1 and op_2 to TM_2. > > > > Originally, one of the primary goals of slot sharing groups was to make > it > > easier for the user to reason about how many slots a job needs > independent > > of the actual number of operators in the job. Interestingly, if all > > operators have their resources properly specified, then slot sharing is > no > > longer needed because Flink could slice off the appropriately sized slots > > for every Task individually. What matters is whether the whole cluster > has > > enough resources to run all tasks or not. > > > > Cheers, > > Till > > > > On Thu, Jan 7, 2021 at 4:08 AM Yangze Guo <karma...@gmail.com> wrote: > > > >> Hi, there, > >> > >> We would like to start a discussion thread on "FLIP-156: Runtime > >> Interfaces for Fine-Grained Resource Requirements"[1], where we > >> propose Slot Sharing Group (SSG) based runtime interfaces for > >> specifying fine-grained resource requirements. > >> > >> In this FLIP: > >> - Expound the user story of fine-grained resource management. > >> - Propose runtime interfaces for specifying SSG-based resource > >> requirements. > >> - Discuss the pros and cons of the three potential granularities for > >> specifying the resource requirements (op, task and slot sharing group) > >> and explain why we choose the slot sharing group. > >> > >> Please find more details in the FLIP wiki document [1]. Looking > >> forward to your feedback. > >> > >> [1] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-156%3A+Runtime+Interfaces+for+Fine-Grained+Resource+Requirements > >> > >> Best, > >> Yangze Guo > >> > >
