One thing which was briefly mentioned in the Flip but not in the implementation plan is the update of the web UI. I think it is worth putting an extra item for updating the web UI to properly display the resources a TM has still to offer with dynamic slot allocation. I guess we need to pull in some JavaScript help in order to implement this step.
Cheers, Till On Mon, Sep 16, 2019 at 2:15 PM Xintong Song <tonysong...@gmail.com> wrote: > Thanks for the comments, Andrey. > > - I agree that instead of ResourceManagerGateway#sendSlotReport, we should > add the default slot resource profile to > ResourceManagerGateway#registerTaskExecutor. > > - If I understand correctly, the reason you suggest do default slot > resource profile first and then do step 3 in a way that support both > TaskExecutorGateway#requestSlot and TaskExecutorGateway#requestResource, is > to try to avoid splitting code paths with the feature option? I think we > can do that, but I also want to bring it up that this can only reduce the > code split by the feature option (which is good) but not eliminate it. We > still need the feature option for the fundamental differences, e.g. > creating new SlotIDs on allocation vs. allocate to free slots with existing > SlotIDs. > > - I don't really think we can do step 5, 6 and 7 independently. Basically > they are all making changes to the same component. We probably can do step > 6 and 7 independently, but I think they both depends on step 5. > > In general, I would say it's good to have as less as possible codes split > by the feature option, which makes the later clean-up easier. But if it > cannot be easily done, I would rather not to put too much efforts on having > a good abstraction and deduplication between the new code path and the > original one that we are removing soon. > > What do you think? > > Thank you~ > > Xintong Song > > > > On Mon, Sep 16, 2019 at 5:59 PM Andrey Zagrebin <and...@ververica.com> > wrote: > > > Hi Xintong, > > > > Thanks for sharing the implementation steps. I also think they makes > sense > > with the feature option. > > > > I was wondering if we could order the steps in a way that each change > does > > not affect other components too much, always having a working system > > then maybe the feature option does not always need to split the code. > Here > > are some thoughts. > > > > - We could do default slot profile firstly and include it into the TM > > registration. I would suggest to add > > to ResourceManagerGateway#registerTaskExecutor, not sendSlotReport. > > This way RM knows about it but does not use at this point. (parts of > step > > 4,6) > > > > - We could try to do step 3 firstly in a way that it also supports the > > current way of allocation in TaskExecutorGateway#requestSlot with the > > default slot profile > > and sends reports both with available resources and with free default > > slots which correspond to the available resources. We can just remove > free > > default slots later. > > The new way of TaskExecutorGateway#requestResource could be also > > implemented here but not used yet. > > > > - Then step 5 can use the new TaskExecutorGateway#requestResource and the > > default slot profile > > > > - Not sure, step 5 and 7 can be implemented independently without > > regression of what we have. Maybe if we do step 7 firstly it will have > only > > default slots firstly and it will simplify step 5 later. > > > > Best, > > Andrey > > > > On Mon, Sep 16, 2019 at 5:53 AM Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > Thanks for the comments, Till and Wenlong. > > > > > > @Wenlong > > > Regarding slot sharing, the general idea is to request a slot with > > > resources for tasks of the entire slot sharing group. Details can be > > found > > > in FLIP-53 [1], regarding how to decide the slot sharing groups and how > > to > > > manage task resources within the shared slots. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Mon, Sep 16, 2019 at 10:42 AM wenlong.lwl <wenlong88....@gmail.com> > > > wrote: > > > > > > > Hi, Xintong, thanks for the great proposal. big +1 for the feature! > It > > is > > > > something like mapreduce-1.0 to mapreduce-2.0. > > > > > > > > I like the design on the whole. One point may need to be included in > > the > > > > proposal：How we deal with slot share group and dynamic slot > allocation? > > > It > > > > can be quite different with dynamic slot allocation. > > > > > > > > On Fri, 13 Sep 2019 at 16:42, Till Rohrmann <trohrm...@apache.org> > > > wrote: > > > > > > > > > Thanks for the update Xintong. From a high level perspective the > > > > > implementation plan looks good to me. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Thu, Sep 12, 2019 at 11:04 AM Xintong Song < > tonysong...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Added implementation steps for this FLIP on the wiki page [1]. > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 20, 2019 at 3:43 PM Xintong Song < > > tonysong...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > @Zili > > > > > > > > > > > > > > As far as I know, Timo is drafting a FLIP that has taken the > > number > > > > 55. > > > > > > > There is a round-up number maintained on the FLIP wiki page [1] > > > shows > > > > > > > which number should be used for the new FLIP, which should be > > > > increased > > > > > > by > > > > > > > whoever takes the number for a new FLIP. > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > > > > > > > > > On Tue, Aug 20, 2019 at 3:28 AM Zili Chen < > wander4...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > >> We suddenly skipped FLIP-55 lol. > > > > > > >> > > > > > > >> > > > > > > >> Xintong Song <tonysong...@gmail.com> 于2019年8月19日周一 下午10:23写道： > > > > > > >> > > > > > > >> > Hi everyone, > > > > > > >> > > > > > > > >> > We would like to start a discussion thread on "FLIP-56: > > Dynamic > > > > Slot > > > > > > >> > Allocation" [1]. This is originally part of the discussion > > > thread > > > > > for > > > > > > >> > "FLIP-53: Fine Grained Resource Management" [2]. As Till > > > > suggested, > > > > > we > > > > > > >> > would like split the original discussion into two topics, > and > > > > start > > > > > a > > > > > > >> > separate new discussion thread as well as FLIP process for > > this > > > > one. > > > > > > >> > > > > > > > >> > Thank you~ > > > > > > >> > > > > > > > >> > Xintong Song > > > > > > >> > > > > > > > >> > > > > > > > >> > [1] > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > > > > > > >> > > > > > > > >> > [2] > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
