Thanks Rui for the comments!
> I have a minor question for this: could we discuss 2 FLIPs together? > I'm afraid the rest endpoints doesn't work as expected when we discuss > WebUI change in the future. > Clarification: Organize related designs in different FLIP wikis and > discuss them in different discussions. Together just means that > two FLIPs can be carried out at the same time. > After the design and discussion of both FLIPs are completed, > we could start voting on each one. It makes sense to me as there are some impacts between the two FLIPs and it could help avoid missing some details. Let’s assume we proceed in this way, What do you think about using FLIP-487 and its corresponding email thread to focus on discussing the UI design to reduce ambiguity? Meanwhile, we can create a new FLIP and email to discuss the features of REST and AdaptiveScheduler, for example, with a title like: "Support AdaptiveScheduler record and query the rescale history," or another more appropriate title. BWT, whatever the final conclusion is, I will adjust the content on the wiki instead of google doc after a reasonable conclusion, and may subsequent discussions and updates be aligned with the FLIP on the wiki. Looking forward to more ideas about it. Best, Yuepeng Pan On 2024/12/06 03:44:24 Rui Fan wrote: > Thanks Yuepeng for driving this proposal! > > It's definitely a great addition for Adaptive Scheduler. > I see FLIP-487 has 2 docs, one is official wiki[1], one is google doc[2]. > It's better to only use one doc to ensure the consistency, I prefer > to only use FLIP-487 official wiki due to you have the wiki permission. > > > I'm also wondering whether it would make sense to focus on the REST > > endpoints in this FLIP and put the UI work in a separate FLIP. WDYT? > > Decreasing the scope would probably help handling the required changes > > Thanks Matthias for the good suggestions, split it to 2 FLIPs makes sense > to me. > I have a minor question for this: could we discuss 2 FLIPs together? > I'm afraid the rest endpoints doesn't work as expected when we discuss > WebUI change in the future. > > Also, I found the current FLIP wiki[1] and doc[2] have deleted all > information > related to WebUI change. If so, the FLIP title should be changed as well. > Currently, the title is "FLIP-487: Show history of rescales in Web UI for > AdaptiveScheduler" > but it seems FLIP-487 doesn't include any Web UI change. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler > [2] > https://docs.google.com/document/d/1WrLBkSkYe2tBQ3j66gKHFr2OB0d1HuHKDrRVr6B8nkM/edit?usp=sharing > > Best, > Rui > > On Wed, Dec 4, 2024 at 9:56 PM Yuepeng Pan <panyuep...@apache.org> wrote: > > > > > Thanks Matthias a lot for the comments and guidance. > > > > >Can we reorganize the draft? Right now, we have some (for RescaleEvent, > > >Required/AcquiredParallelism) schema defined in the "Proposed Changes" > > section > > >and some other schema under "Public Interfaces". It would be nice to have > > this > > >more organized. Just as a suggestion: In the end the proposed changes > > should list > > >the different REST endpoints you want to introduce (including the > > corresponding > > >schemas for request and response). > > > > Thanks for the comments. I’ve made some structural adjustments and content > > modifications to help clarify certain issues. Since some questions are > > still under discussion, I plan to update the document once we reach a > > consensus. > > > > > I'm also wondering whether it would make sense to focus on the REST > > > endpoints in this FLIP and put the UI work in a separate FLIP. WDYT? > > Decreasing > > > the scope would probably help handling the required changes. > > > > Nice proposal ! That will help us better focus our attention and > > concentrate on the main target. > > > > > Have you considered adding the onChange event timestamp for a rescale > > event > > > as well? We introduced a separation of the job requirements change event > > > and the actual rescale execution in FLIP-461 [1]. It might be worth > > > documenting the time when a change was monitored for the first time that > > > triggered the rescale. WDYT? > > > > Thank you very much for your comments.As described in the document, the > > original design had many abstract concepts that were relatively simple. The > > information recorded for each rescale event mainly reflected the current > > state, rather than the entire historical state of the adaptive scheduler > > throughout the rescale process. > > IIUC, when we add the onChange event timestamp for rescale events, each > > rescale event will capture the state changes of the adaptive scheduler at > > various stages (these states would form a collection similar to the set of > > all possible states of the adaptive scheduler) along with the corresponding > > timestamps. These states would essentially form an ordered collection. > > Compared to the original design, this would undoubtedly make internal > > information more transparent, which I think is very meaningful. > > If I misunderstood, please correct me. > > > > >You're mentioning "comments" as a field of the RescaleEvent in your > > proposal. > > >What's the use-case here? Where are these comments from? (update) A brief > > talk > > >with Yuepeng on that topic revealed that the field is supposed to be used > > for > > >errors that occurred during the rescale operation. My take on that one: > > >- We might want to reconsider the field name in that case (maybe > > >errors_during_rescale?). "comments" seems to be quite generic. > > > - Additionally, shouldn't we make this a list of errors rather than a > > String field? > > > > Sorry for not fully explaining the purpose of this field earlier. The > > intention behind adding this field is to highlight special circumstances > > during a rescale, such as exceptions caused by insufficient resources or > > situations where a new request overrides the current one, forcing us to > > abandon the ongoing rescale operation and reallocate resources based on the > > new request. In such cases, it would be valuable to record what exactly > > happened internally. That said, I completely agree with the point that > > "comments" might seem overly broad. > > Maybe we need to address a few key questions here: > > > > 1. Is it necessary to retain a "comments"-like field to capture such > > information? > > 2. Should error-related information also be retained? > > 3. If retained, where should the "comments" (if applicable) and error > > information fields be located? > > - Should they be tied to a specific adaptive-scheduler-state window? > > - Or should they be directly included in the rescale event message? > > 4. Should these fields be implemented as lists? > > > > For the first and second items, I prefer to retain both types of > > information because they can provide transparent details to the user. > > For the third and fourth items, I prefer to associate these two fields as > > lists with specific adaptive-scheduler state windows. This approach could > > help organize the information better and link it to relevant events in the > > rescale process. > > > > I would greatly appreciate hearing more suggestions and ideas! > > > > > - How certain are we that we can associate errors to the actual rescale > > > operation and rather than the error being caused by something else? > > > > In my limited reading, it seems that only exceptions related to resource > > shortage during parallelism allocation can be classified into this type of > > exception. Other exceptions can be categorized outside this scope. > > > > > But there is no "attempt number" mentioned in the RescaleEvent schema. > > > > Please let me try to add to this question: > > - Each rescale requirement corresponds to a resourceRequirementsEpochID. > > - For the same rescale requirement (resourceRequirementsEpochID), multiple > > rescale attempts may occur to here. We treat each rescale under the current > > resource requirements as a rescale attempt. That is, for each rescale > > attempt on the same rescale requirement (resourceRequirementsEpochID), the > > rescale attempt id will increment by 1. When the > > resourceRequirementsEpochID is changed, the rescale attempt will be reset. > > - Based on these two points, whenever a rescale attempt happens, the > > rescaleID will increment globally. > > > > I hope this may clarify the logic and approach. Please let me know what’s > > your opinion. > > > > > Additionally, what is the ID based on? Do we start from 0 and just > > increment? > > > Or do we want to have a mechanism that ensures that the IDs are also > > > unique/monotonically increasing after JobManager failovers? > > > > To be honest, I expect initially it to increment from 0 every time a job > > is started since the rescale ID is not strongly tied to the state. If we > > can ensure that the ID remains unique and increase monotonically even after > > a JobManager failover, this would make the rescale ID more consistent, > > right? If so, that makes sense to me. Please let me know your thoughts. > > > > > For the parallelism schema: I might be misreading the draft here but > > you're proposing to use the subtask > > > name as the ID to refer to the JobVertex? That the name might become > > quite long. What about using the > > > JobVertexID here. That would be also more aligned to how the parallelism > > is represented by the > > > /jobs/<job-id>/resource-requirements endpoint. If we want to add the > > task name for readability purposes, we > > > can still add this one as a taskName field to the > > Required/AcquiredParallelism schema. > > > > Good proposal! I did consider using JobVertexID, but for ease of use, I > > ended up changing it to taskName. I'm wondering if we could use both > > JobVertexID and taskName. Could we provide a parameter to control the > > maximum length of taskName? In this way, we could ensure the uniqueness of > > JobVertexID while maintaining a certain level of readability for taskName. > > WDYTA? > > > > > Status field: > > > - What is the meaning of "TRYING"? I guess, we're more or less using the > > > AdaptiveScheduler states here, aren't we? Can't we align/stick to the > > > naming that's defined in the AdaptiveScheduler state? > > > > As mentioned above, this type of state in the design is used to describe > > the current status of a rescale event. It is not equivalent to the state of > > the adaptive scheduler; rather, it represents a window just after receiving > > an update request but before reaching the next defined state. > > > > > Can't we align/stick to the naming that's defined in the > > AdaptiveScheduler state? > > > > I previously conducted some research on the states of the > > AdaptiveScheduler [1]. Plase let me first try to define the necessary > > conditions for a Rescale Event: > > > > - The process starting from WaitingForResources / CreatingExecutionGraph > > and ending at Executing / FINISHED is considered one rescale. > > - If the process starts from WaitingForResources / CreatingExecutionGraph > > and ends at Executing, this can be considered a successful rescale. > > - If the process starts from WaitingForResources / CreatingExecutionGraph > > and transitions directly to FINISHED for other reasons, it can be > > considered a failed rescale. > > Based on these definitions, the adaptive-scheduler states that can be > > included in the rescale process seem to be Created, CreatingExecutionGraph, > > WaitingForResources, Executing, StopWithSavepoint, and Restarting. > > > > If everything is aligned and we treat the AdaptiveScheduler states as > > equivalent to the RescaleEvent states, some of the states might seem a bit > > odd. For example, after a successful rescale, the final state of the > > rescale event would be Executing(as same as adaptive scheduler). Would we > > consider this situation reasonable? If so, Aligning the states would make > > sense to me. > > Additionally, if the above definition holds, each rescale would need a > > "final state" or “current state” field to indicate its status. > > > > Maybe it’s important to define and reach consensus on the rescale event > > and its start and end boundaries. > > Please correct me if I’m wrong. Thank you very much! > > > > > > > Do we really need a new REST endpoint for the configuration? Can't we > > get > > > the provided information already from the existing configuration > > endpoint? > > > > Thank you for your comments. Sorry for not finding a suitable existing > > REST interface to reuse. > > But there’s a part of ‘/jobs/:jobid/config‘ that could be reused at the > > backend side. > > Another reason for introducing a new interface is to directly[2] retrieve > > the latest values from the Adaptive Scheduler, as we cannot guarantee that > > the configuration values used in the Adaptive Scheduler are always the ones > > specified in the configuration file. For example, some parameter values may > > be overwritten, which could change them. WDYTA? > > > > > For the summary endpoint: I see similarities to the checkpoint summary > > > here. Not sure whether you already considered that but would it make > > sense > > > to align the field names in some way to have a consistent look-and-feel? > > > I'm also wondering whether it makes sense to align the schema to have > > > something like the latest rescale, failed rescale, ... > > > > Thank you for the reminder. Yes, I have considered it. For example, I > > tried to align the request information in the REST interface with the > > checkpoint summary, as seen in this document[2]. However, I haven't > > strictly followed the same interface style and fields as the checkpoint > > REST interface, and instead used a simpler list (sorry, I should have > > shared these considerations in the draft earlier to give developers more > > context). If needed, I think it’s a great choice. > > > > By the way, perhaps we should also consider the archive feature for the > > rescale history, to support viewing this information on the history server. > > > > Thank you very much. > > > > > > [1] > > https://drive.google.com/file/d/1FE17cAfbUXypYBp0soaVLu5oqn39BoKI/view?usp=sharing > > [2] > > https://docs.google.com/document/d/1WrLBkSkYe2tBQ3j66gKHFr2OB0d1HuHKDrRVr6B8nkM/edit?disco=AAABSsVvI6Q > > > > > > > > > > On 2024/12/02 08:59:16 Matthias Pohl wrote: > > > Hi Yuepeng, > > > thanks for the proposal. Having a way to see the history of rescales is a > > > nice feature, I guess. I went over the draft and have a few questions: > > > > > > Can we reorganize the draft? Right now, we have some (for RescaleEvent, > > > Required/AcquiredParallelism) schema defined in the "Proposed Changes" > > > section and some other schema under "Public Interfaces". It would be nice > > > to have this more organized. > > > Just as a suggestion: In the end the proposed changes should list the > > > different REST endpoints you want to introduce (including the > > corresponding > > > schemas for request and response). > > > --- > > > I'm also wondering whether it would make sense to focus on the REST > > > endpoints in this FLIP and put the UI work in a separate FLIP. WDYT? > > > Decreasing the scope would probably help handling the required changes. > > > --- > > > Have you considered adding the onChange event timestamp for a rescale > > event > > > as well? We introduced a separation of the job requirements change event > > > and the actual rescale execution in FLIP-461 [1]. It might be worth > > > documenting the time when a change was monitored for the first time that > > > triggered the rescale. WDYT? > > > --- > > > You're mentioning "comments" as a field of the RescaleEvent in your > > > proposal. What's the use-case here? Where are these comments from? > > > > > > (update) > > > A brief talk with Yuepeng on that topic revealed that the field is > > supposed > > > to be used for errors that occurred during the rescale operation. My take > > > on that one: > > > - We might want to reconsider the field name in that case (maybe > > > errors_during_rescale?). "comments" seems to be quite generic. > > > - Additionally, shouldn't we make this a list of errors rather than a > > > String field? > > > - How certain are we that we can associate errors to the actual rescale > > > operation and rather than the error being caused by something else? > > > --- > > > In the schema of the RescaleEvent you describe the three different > > > ID/numbers in the following way: > > > > > > > The ‘id’ is automatically incremental, The rescaleAttemptId is > > generated > > > > based on one specified resource-requirement and the attempt number is > > > > generated based on rescaleAttemptId. > > > > > > But there is no "attempt number" mentioned in the RescaleEvent schema. > > > Additionally, what is the ID based on? Do we start from 0 and just > > > increment? Or do we want to have a mechanism that ensures that the IDs > > are > > > also unique/monotonically increasing after JobManager failovers? > > > --- > > > For the parallelism schema: I might be misreading the draft here but > > you're > > > proposing to use the subtask name as the ID to refer to the JobVertex? > > That > > > the name might become quite long. What about using the JobVertexID here. > > > That would be also more aligned to how the parallelism is represented by > > > the /jobs/<job-id>/resource-requirements endpoint. If we want to add the > > > task name for readability purposes, we can still add this one as a > > taskName > > > field to the Required/AcquiredParallelism schema. > > > --- > > > Status field: > > > - What is the meaning of "TRYING"? I guess, we're more or less using the > > > AdaptiveScheduler states here, aren't we? Can't we align/stick to the > > > naming that's defined in the AdaptiveScheduler state? > > > --- > > > Do we really need a new REST endpoint for the configuration? Can't we get > > > the provided information already from the existing configuration > > endpoint? > > > That said, I still find it useful to have a config tab in the UI at the > > end. > > > --- > > > For the summary endpoint: I see similarities to the checkpoint summary > > > here. Not sure whether you already considered that but would it make > > sense > > > to align the field names in some way to have a consistent look-and-feel? > > > I'm also wondering whether it makes sense to align the schema to have > > > something like latest rescale, failed rescale, ... > > > > > > Best, > > > Matthias > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing+for+the+AdaptiveScheduler > > > > > > On Mon, Nov 25, 2024 at 11:24 AM yuanfeng hu <yuanf...@apache.org> > > wrote: > > > > > > > +1, I think this feature is very useful for adaptive scheduler. > > > > > > > > Yuepeng Pan <panyuep...@apache.org> 于2024年11月22日周五 18:38写道： > > > > > > > > > Hi community, > > > > > > > > > > > > > > > > > > > > > > > > > Currently, the Adaptive Scheduler already supports the REST API > > > > > > > > > > to manually adjust[1] the parallelism of jobs, which enhances the > > > > > > > > > > functionality of the Adaptive Scheduler. > > > > > > > > > > However, Adaptive Scheduler doesn't support displaying or tracing the > > > > > rescale history yet[2]. > > > > > > > > > > This makes it inconvenient for users/devs to quickly obtain some > > internal > > > > > > > > > > information about the rescale history of the Adaptive Scheduler. > > > > > > > > > > And showing the history of rescale events of AdaptiveScheduler in > > the web > > > > > > > > > > UI is very useful for users to make the next step for jobs. > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, I created the FLIP-487[3] doc to support > > > > > > > > > > 'Show history of rescales in Web UI for AdaptiveScheduler'. > > > > > > > > > > Please refer to the google document[3] for more details > > > > > > > > > > about the proposed design and implementation. > > > > > > > > > > > > > > > > > > > > > > > > > Looking forward to any feedback and opinions on this proposal. > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management > > > > > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-22258 > > > > > > > > > > [3] > > > > > > > > > > > https://docs.google.com/document/d/1WrLBkSkYe2tBQ3j66gKHFr2OB0d1HuHKDrRVr6B8nkM/edit?tab=t.0 > > > > > > > > > > > > > > > > > > > > > > > > > Thank you very much. > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Regards. > > > > > > > > > > Yuepeng Pan > > > > > > > > > > > > > > > > -- > > > > Best, > > > > Yuanfeng > > > > > > > > > >
