> I suppose we could further remove the min because it would always be safer to scale down if resources are not available than not to run at all [1].
Apart from what @Roman has already mentioned, there are still cases where we're certain that there is no point in running the jobs with resources lower than X; e.g., because the state is too large to be processed with parallelism of 1; this allows you not to waste resources if you're certain that the job would go into the restart loop / won't be able to checkpoint I believe that for most use cases, simply keeping the lower bound at 1 will be sufficient. > I saw that the minimum bound is currently not used in the code you posted above [2]. Is that still planned? Yes. We already allow setting the lower bound via API, but it's not considered by the scheduler. I'll address this limitation in a separate issue. > Note that originally we had assumed min == max but I think that would be a less safe scaling approach because we would get stuck waiting for resources when they are not available, e.g. k8s resource limits reached. 100% agreed; The above-mentioned knobs should allow you to balance the trade-off. Does that make sense? Best, D. On Tue, Feb 28, 2023 at 1:14 PM Roman Khachatryan <ro...@apache.org> wrote: > Hi, > > Thanks for the update, I think distinguishing the rescaling behaviour and > the desired parallelism declaration is important. > > Having the ability to specify min parallelism might be useful in > environments with multiple jobs: Scheduler will then have an option to stop > the less suitable job. > In other setups, where the job should not be stopped at all, the user can > always set it to 0. > > Regards, > Roman > > > On Tue, Feb 28, 2023 at 12:58 PM Maximilian Michels <m...@apache.org> > wrote: > >> Hi David, >> >> Thanks for the update! We consider using the new declarative resource >> API for autoscaling. Currently, we treat a scaling decision as a new >> deployment which means surrendering all resources to Kubernetes and >> subsequently reallocating them for the rescaled deployment. The >> declarative resource management API is a great step forward because it >> allows us to do faster and safer rescaling. Faster, because we can >> continue to run while resources are pre-allocated which minimizes >> downtime. Safer, because we can't get stuck when the desired resources >> are not available. >> >> An example with two vertices and their respective parallelisms: >> v1: 50 >> v2: 10 >> Let's assume slot sharing is disabled, so we need 60 task slots to run >> the vertices. >> >> If the autoscaler was to decide to scale up v1 and v2, it could do so >> in a safe way by using min/max configuration: >> v1: [min: 50, max: 70] >> v2: [min: 10, max: 20] >> This would then need 90 task slots to run at max capacity. >> >> I suppose we could further remove the min because it would always be >> safer to scale down if resources are not available than to not run at >> all [1]. In fact, I saw that the minimum bound is currently not used >> in the code you posted above [2]. Is that still planned? >> >> -Max >> >> PS: Note that originally we had assumed min == max but I think that >> would be a less safe scaling approach because we would get stuck >> waiting for resources when they are not available, e.g. k8s resource >> limits reached. >> >> [1] However, there might be costs involved with executing the >> rescaling, e.g. for using external storage like s3, especially without >> local recovery. >> [2] >> https://github.com/dmvk/flink/commit/5e7edcb77d8522c367bc6977f80173b14dc03ce9 >> >> On Tue, Feb 28, 2023 at 9:33 AM David Morávek <d...@apache.org> wrote: >> > >> > Hi Everyone, >> > >> > We had some more talks about the pre-allocation of resources with @Max, >> and >> > here is the final state that we've converged to for now: >> > >> > The vital thing to note about the new API is that it's declarative, >> meaning >> > we're declaring the desired state to which we want our job to converge; >> If, >> > after the requirements update job no longer holds the desired resources >> > (fewer resources than the lower bound), it will be canceled and >> transition >> > back into the waiting for resources state. >> > >> > In some use cases, you might always want to rescale to the upper bound >> > (this goes along the lines of "preallocating resources" and minimizing >> the >> > number of rescales, which is especially useful with the large state). >> This >> > can be controlled by two knobs that already exist: >> > >> > 1) "jobmanager.adaptive-scheduler.min-parallelism-increase" - this >> affects >> > a minimal parallelism increase step of a running job; we'll slightly >> change >> > the semantics, and we'll trigger rescaling either once this condition is >> > met or when you hit the ceiling; setting this to the high number will >> > ensure that you always rescale to the upper bound >> > >> > 2) "jobmanager.adaptive-scheduler.resource-stabilization-timeout" - for >> new >> > and already restarting jobs, we'll always respect this timeout, which >> > allows you to wait for more resources even though you already have more >> > resources than defined in the lower bound; again, in the case we reach >> the >> > ceiling (the upper bound), we'll transition into the executing state. >> > >> > >> > We're still planning to dig deeper in this direction with other efforts, >> > but this is already good enough and should allow us to move the FLIP >> > forward. >> > >> > WDYT? Unless there are any objectives against the above, I'd like to >> > proceed to a vote. >> > >> > Best, >> > D. >> > >> > On Thu, Feb 23, 2023 at 5:39 PM David Morávek <d...@apache.org> wrote: >> > >> > > Hi Everyone, >> > > >> > > @John >> > > >> > > This is a problem that we've spent some time trying to crack; in the >> end, >> > > we've decided to go against doing any upgrades to JobGraphStore from >> > > JobMaster to avoid having multiple writers that are guarded by >> different >> > > leader election lock (Dispatcher and JobMaster might live in a >> different >> > > process). The contract we've decided to choose instead is leveraging >> the >> > > idempotency of the endpoint and having the user of the API retry in >> case >> > > we're unable to persist new requirements in the JobGraphStore [1]. We >> > > eventually need to move JobGraphStore out of the dispatcher, but >> that's way >> > > out of the scope of this FLIP. The solution is a deliberate >> trade-off. The >> > > worst scenario is that the Dispatcher fails over in between retries, >> which >> > > would simply rescale the job to meet the previous resource >> requirements >> > > (more extended unavailability of underlying HA storage would have >> worse >> > > consequences than this). Does that answer your question? >> > > >> > > @Matthias >> > > >> > > Good catch! I'm fixing it now, thanks! >> > > >> > > [1] >> > > >> https://github.com/dmvk/flink/commit/5e7edcb77d8522c367bc6977f80173b14dc03ce9#diff-a4b690fb2c4975d25b05eb4161617af0d704a85ff7b1cad19d3c817c12f1e29cR1151 >> > > >> > > Best, >> > > D. >> > > >> > > On Tue, Feb 21, 2023 at 12:24 AM John Roesler <vvcep...@apache.org> >> wrote: >> > > >> > >> Thanks for the FLIP, David! >> > >> >> > >> I just had one small question. IIUC, the REST API PUT request will go >> > >> through the new DispatcherGateway method to be handled. Then, after >> > >> validation, the dispatcher would call the new JobMasterGateway >> method to >> > >> actually update the job. >> > >> >> > >> Which component will write the updated JobGraph? I just wanted to >> make >> > >> sure it’s the JobMaster because it it were the dispatcher, there >> could be a >> > >> race condition with the async JobMaster method. >> > >> >> > >> Thanks! >> > >> -John >> > >> >> > >> On Mon, Feb 20, 2023, at 07:34, Matthias Pohl wrote: >> > >> > Thanks for your clarifications, David. I don't have any additional >> major >> > >> > points to add. One thing about the FLIP: The RPC layer API for >> updating >> > >> the >> > >> > JRR returns a future with a JRR? I don't see value in returning a >> JRR >> > >> here >> > >> > since it's an idempotent operation? Wouldn't it be enough to return >> > >> > CompletableFuture<Void> here? Or am I missing something? >> > >> > >> > >> > Matthias >> > >> > >> > >> > On Mon, Feb 20, 2023 at 1:48 PM Maximilian Michels <m...@apache.org >> > >> > >> wrote: >> > >> > >> > >> >> Thanks David! If we could get the pre-allocation working as part >> of >> > >> >> the FLIP, that would be great. >> > >> >> >> > >> >> Concerning the downscale case, I agree this is a special case for >> the >> > >> >> (single-job) application mode where we could re-allocate slots in >> a >> > >> >> way that could leave entire task managers unoccupied which we >> would >> > >> >> then be able to release. The goal essentially is to reduce slot >> > >> >> fragmentation on scale down by packing the slots efficiently. The >> > >> >> easiest way to add this optimization when running in application >> mode >> > >> >> would be to drop as many task managers during the restart such >> that >> > >> >> NUM_REQUIRED_SLOTS >= NUM_AVAILABLE_SLOTS stays true. We can look >> into >> > >> >> this independently of the FLIP. >> > >> >> >> > >> >> Feel free to start the vote. >> > >> >> >> > >> >> -Max >> > >> >> >> > >> >> On Mon, Feb 20, 2023 at 9:10 AM David Morávek <d...@apache.org> >> wrote: >> > >> >> > >> > >> >> > Hi everyone, >> > >> >> > >> > >> >> > Thanks for the feedback! I've updated the FLIP to use >> idempotent PUT >> > >> API >> > >> >> instead of PATCH and to properly handle lower bound settings, to >> > >> support >> > >> >> the "pre-allocation" of the resources. >> > >> >> > >> > >> >> > @Max >> > >> >> > >> > >> >> > > How hard would it be to address this issue in the FLIP? >> > >> >> > >> > >> >> > I've included this in the FLIP. It might not be too hard to >> implement >> > >> >> this in the end. >> > >> >> > >> > >> >> > > B) drop as many superfluous task managers as needed >> > >> >> > >> > >> >> > I've intentionally left this part out for now because this >> ultimately >> > >> >> needs to be the responsibility of the Resource Manager. After >> all, in >> > >> the >> > >> >> Session Cluster scenario, the Scheduler doesn't have the bigger >> > >> picture of >> > >> >> other tasks of other jobs running on those TMs. This will most >> likely >> > >> be a >> > >> >> topic for another FLIP. >> > >> >> > >> > >> >> > WDYT? If there are no other questions or concerns, I'd like to >> start >> > >> the >> > >> >> vote on Wednesday. >> > >> >> > >> > >> >> > Best, >> > >> >> > D. >> > >> >> > >> > >> >> > On Wed, Feb 15, 2023 at 3:34 PM Maximilian Michels < >> m...@apache.org> >> > >> >> wrote: >> > >> >> >> >> > >> >> >> I missed that the FLIP states: >> > >> >> >> >> > >> >> >> > Currently, even though we’d expose the lower bound for >> clarity and >> > >> >> API completeness, we won’t allow setting it to any other value >> than one >> > >> >> until we have full support throughout the stack. >> > >> >> >> >> > >> >> >> How hard would it be to address this issue in the FLIP? >> > >> >> >> >> > >> >> >> There is not much value to offer setting a lower bound which >> won't >> > >> be >> > >> >> >> respected / throw an error when it is set. If we had support >> for a >> > >> >> >> lower bound, we could enforce a resource contract externally >> via >> > >> >> >> setting lowerBound == upperBound. That ties back to the >> Rescale API >> > >> >> >> discussion we had. I want to better understand what the major >> > >> concerns >> > >> >> >> would be around allowing this. >> > >> >> >> >> > >> >> >> Just to outline how I imagine the logic to work: >> > >> >> >> >> > >> >> >> A) The resource constraints are already met => Nothing changes >> > >> >> >> B) More resources available than required => Cancel the job, >> drop as >> > >> >> >> many superfluous task managers as needed, restart the job >> > >> >> >> C) Less resources available than required => Acquire new task >> > >> >> >> managers, wait for them to register, cancel and restart the job >> > >> >> >> >> > >> >> >> I'm open to helping out with the implementation. >> > >> >> >> >> > >> >> >> -Max >> > >> >> >> >> > >> >> >> On Mon, Feb 13, 2023 at 7:45 PM Maximilian Michels < >> m...@apache.org> >> > >> >> wrote: >> > >> >> >> > >> > >> >> >> > Based on further discussion I had with Chesnay on this PR >> [1], I >> > >> think >> > >> >> >> > jobs would currently go into a restarting state after the >> resource >> > >> >> >> > requirements have changed. This wouldn't achieve what we had >> in >> > >> mind, >> > >> >> >> > i.e. sticking to the old resource requirements until enough >> slots >> > >> are >> > >> >> >> > available to fulfil the new resource requirements. So this >> may >> > >> not be >> > >> >> >> > 100% what we need but it could be extended to do what we >> want. >> > >> >> >> > >> > >> >> >> > -Max >> > >> >> >> > >> > >> >> >> > [1] >> > >> https://github.com/apache/flink/pull/21908#discussion_r1104792362 >> > >> >> >> > >> > >> >> >> > On Mon, Feb 13, 2023 at 7:16 PM Maximilian Michels < >> > >> m...@apache.org> >> > >> >> wrote: >> > >> >> >> > > >> > >> >> >> > > Hi David, >> > >> >> >> > > >> > >> >> >> > > This is awesome! Great writeup and demo. This is pretty >> much >> > >> what we >> > >> >> >> > > need for the autoscaler as part of the Flink Kubernetes >> operator >> > >> >> [1]. >> > >> >> >> > > Scaling Flink jobs effectively is hard but fortunately we >> have >> > >> >> solved >> > >> >> >> > > the issue as part of the Flink Kubernetes operator. The >> only >> > >> >> critical >> > >> >> >> > > piece we are missing is a better way to execute scaling >> > >> decisions, >> > >> >> as >> > >> >> >> > > discussed in [2]. >> > >> >> >> > > >> > >> >> >> > > Looking at your proposal, we would set lowerBound == >> upperBound >> > >> for >> > >> >> >> > > the parallelism because we want to fully determine the >> > >> parallelism >> > >> >> >> > > externally based on the scaling metrics. Does that sound >> right? >> > >> >> >> > > >> > >> >> >> > > What is the timeline for these changes? Is there a JIRA? >> > >> >> >> > > >> > >> >> >> > > Cheers, >> > >> >> >> > > Max >> > >> >> >> > > >> > >> >> >> > > [1] >> > >> >> >> > >> >> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ >> > >> >> >> > > [2] >> > >> >> https://lists.apache.org/thread/2f7dgr88xtbmsohtr0f6wmsvw8sw04f5 >> > >> >> >> > > >> > >> >> >> > > On Mon, Feb 13, 2023 at 1:16 PM feng xiangyu < >> > >> xiangyu...@gmail.com> >> > >> >> wrote: >> > >> >> >> > > > >> > >> >> >> > > > Hi David, >> > >> >> >> > > > >> > >> >> >> > > > Thanks for your reply. I think your response totally >> make >> > >> >> sense. This >> > >> >> >> > > > flip targets on declaring required resource to >> ResourceManager >> > >> >> instead of >> > >> >> >> > > > using ResourceManager to add/remove TMs directly. >> > >> >> >> > > > >> > >> >> >> > > > Best, >> > >> >> >> > > > Xiangyu >> > >> >> >> > > > >> > >> >> >> > > > >> > >> >> >> > > > >> > >> >> >> > > > David Morávek <david.mora...@gmail.com> 于2023年2月13日周一 >> > >> 15:46写道： >> > >> >> >> > > > >> > >> >> >> > > > > Hi everyone, >> > >> >> >> > > > > >> > >> >> >> > > > > @Shammon >> > >> >> >> > > > > >> > >> >> >> > > > > I'm not entirely sure what "config file" you're >> referring >> > >> to. >> > >> >> You can, of >> > >> >> >> > > > > course, override the default parallelism in >> > >> "flink-conf.yaml", >> > >> >> but for >> > >> >> >> > > > > sinks and sources, the parallelism needs to be tweaked >> on >> > >> the >> > >> >> connector >> > >> >> >> > > > > level ("WITH" statement). >> > >> >> >> > > > > >> > >> >> >> > > > > This is something that should be achieved with tooling >> > >> around >> > >> >> Flink. We >> > >> >> >> > > > > want to provide an API on the lowest level that >> generalizes >> > >> >> well. Achieving >> > >> >> >> > > > > what you're describing should be straightforward with >> this >> > >> API. >> > >> >> >> > > > > >> > >> >> >> > > > > @Xiangyu >> > >> >> >> > > > > >> > >> >> >> > > > > Is it possible for this REST API to declare TM >> resources in >> > >> the >> > >> >> future? >> > >> >> >> > > > > >> > >> >> >> > > > > >> > >> >> >> > > > > Would you like to add/remove TMs if you use an active >> > >> Resource >> > >> >> Manager? >> > >> >> >> > > > > This would be out of the scope of this effort since it >> > >> targets >> > >> >> the >> > >> >> >> > > > > scheduler component only (we make no assumptions about >> the >> > >> used >> > >> >> Resource >> > >> >> >> > > > > Manager). Also, the AdaptiveScheduler is only intended >> to be >> > >> >> used for >> > >> >> >> > > > > Streaming. >> > >> >> >> > > > > >> > >> >> >> > > > > And for streaming jobs, I'm wondering if there is any >> > >> >> situation we need to >> > >> >> >> > > > > > rescale the TM resources of a flink cluster at first >> and >> > >> then >> > >> >> the >> > >> >> >> > > > > adaptive >> > >> >> >> > > > > > scheduler will rescale the per-vertex >> ResourceProfiles >> > >> >> accordingly. >> > >> >> >> > > > > > >> > >> >> >> > > > > >> > >> >> >> > > > > We plan on adding support for the ResourceProfiles >> (dynamic >> > >> slot >> > >> >> >> > > > > allocation) as the next step. Again we won't make any >> > >> >> assumptions about the >> > >> >> >> > > > > used Resource Manager. In other words, this effort >> ends by >> > >> >> declaring >> > >> >> >> > > > > desired resources to the Resource Manager. >> > >> >> >> > > > > >> > >> >> >> > > > > Does that make sense? >> > >> >> >> > > > > >> > >> >> >> > > > > @Matthias >> > >> >> >> > > > > >> > >> >> >> > > > > We've done another pass on the proposed API and >> currently >> > >> lean >> > >> >> towards >> > >> >> >> > > > > having an idempotent PUT API. >> > >> >> >> > > > > - We don't care too much about multiple writers' >> scenarios >> > >> in >> > >> >> terms of who >> > >> >> >> > > > > can write an authoritative payload; this is up to the >> user >> > >> of >> > >> >> the API to >> > >> >> >> > > > > figure out >> > >> >> >> > > > > - It's indeed tricky to achieve atomicity with PATCH >> API; >> > >> >> switching to PUT >> > >> >> >> > > > > API seems to do the trick >> > >> >> >> > > > > - We won't allow partial "payloads" anymore, meaning >> you >> > >> need >> > >> >> to define >> > >> >> >> > > > > requirements for all vertices in the JobGraph; This is >> > >> >> completely fine for >> > >> >> >> > > > > the programmatic workflows. For DEBUG / DEMO purposes, >> you >> > >> can >> > >> >> use the GET >> > >> >> >> > > > > endpoint and tweak the response to avoid writing the >> whole >> > >> >> payload by hand. >> > >> >> >> > > > > >> > >> >> >> > > > > WDYT? >> > >> >> >> > > > > >> > >> >> >> > > > > >> > >> >> >> > > > > Best, >> > >> >> >> > > > > D. >> > >> >> >> > > > > >> > >> >> >> > > > > On Fri, Feb 10, 2023 at 11:21 AM feng xiangyu < >> > >> >> xiangyu...@gmail.com> >> > >> >> >> > > > > wrote: >> > >> >> >> > > > > >> > >> >> >> > > > > > Hi David, >> > >> >> >> > > > > > >> > >> >> >> > > > > > Thanks for creating this flip. I think this work it >> is >> > >> very >> > >> >> useful, >> > >> >> >> > > > > > especially in autoscaling scenario. I would like to >> share >> > >> >> some questions >> > >> >> >> > > > > > from my view. >> > >> >> >> > > > > > >> > >> >> >> > > > > > 1, Is it possible for this REST API to declare TM >> > >> resources >> > >> >> in the >> > >> >> >> > > > > future? >> > >> >> >> > > > > > I'm asking because we are building the autoscaling >> feature >> > >> >> for Flink OLAP >> > >> >> >> > > > > > Session Cluster in ByteDance. We need to rescale the >> > >> >> cluster's resource >> > >> >> >> > > > > on >> > >> >> >> > > > > > TM level instead of Job level. It would be very >> helpful >> > >> if we >> > >> >> have a REST >> > >> >> >> > > > > > API for out external Autoscaling service to use. >> > >> >> >> > > > > > >> > >> >> >> > > > > > 2, And for streaming jobs, I'm wondering if there is >> any >> > >> >> situation we >> > >> >> >> > > > > need >> > >> >> >> > > > > > to rescale the TM resources of a flink cluster at >> first >> > >> and >> > >> >> then the >> > >> >> >> > > > > > adaptive scheduler will rescale the per-vertex >> > >> >> ResourceProfiles >> > >> >> >> > > > > > accordingly. >> > >> >> >> > > > > > >> > >> >> >> > > > > > best. >> > >> >> >> > > > > > Xiangyu >> > >> >> >> > > > > > >> > >> >> >> > > > > > Shammon FY <zjur...@gmail.com> 于2023年2月9日周四 11:31写道： >> > >> >> >> > > > > > >> > >> >> >> > > > > > > Hi David >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > Thanks for your answer. >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > > Can you elaborate more about how you'd intend to >> use >> > >> the >> > >> >> endpoint? I >> > >> >> >> > > > > > > think we can ultimately introduce a way of >> re-declaring >> > >> >> "per-vertex >> > >> >> >> > > > > > > defaults," but I'd like to understand the use case >> bit >> > >> more >> > >> >> first. >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > For this issue, I mainly consider the consistency >> of >> > >> user >> > >> >> configuration >> > >> >> >> > > > > > and >> > >> >> >> > > > > > > job runtime. For sql jobs, users usually set >> specific >> > >> >> parallelism for >> > >> >> >> > > > > > > source and sink, and set a global parallelism for >> other >> > >> >> operators. >> > >> >> >> > > > > These >> > >> >> >> > > > > > > config items are stored in a config file. For some >> > >> >> high-priority jobs, >> > >> >> >> > > > > > > users may want to manage them manually. >> > >> >> >> > > > > > > 1. When users need to scale the parallelism, they >> should >> > >> >> update the >> > >> >> >> > > > > > config >> > >> >> >> > > > > > > file and restart flink job, which may take a long >> time. >> > >> >> >> > > > > > > 2. After providing the REST API, users can just >> send a >> > >> >> request to the >> > >> >> >> > > > > job >> > >> >> >> > > > > > > via REST API quickly after updating the config >> file. >> > >> >> >> > > > > > > The configuration in the running job and config >> file >> > >> should >> > >> >> be the >> > >> >> >> > > > > same. >> > >> >> >> > > > > > > What do you think of this? >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > best. >> > >> >> >> > > > > > > Shammon >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > On Tue, Feb 7, 2023 at 4:51 PM David Morávek < >> > >> >> david.mora...@gmail.com> >> > >> >> >> > > > > > > wrote: >> > >> >> >> > > > > > > >> > >> >> >> > > > > > > > Hi everyone, >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Let's try to answer the questions one by one. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > *@ConradJam* >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > when the number of "slots" is insufficient, can >> we can >> > >> >> stop users >> > >> >> >> > > > > > > rescaling >> > >> >> >> > > > > > > > > or throw something to tell user "less avaliable >> > >> slots >> > >> >> to upgrade, >> > >> >> >> > > > > > > please >> > >> >> >> > > > > > > > > checkout your alivalbe slots" ? >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > The main property of AdaptiveScheduler is that >> it can >> > >> >> adapt to >> > >> >> >> > > > > > "available >> > >> >> >> > > > > > > > resources," which means you're still able to make >> > >> >> progress even >> > >> >> >> > > > > though >> > >> >> >> > > > > > > you >> > >> >> >> > > > > > > > didn't get all the slots you've asked for. Let's >> break >> > >> >> down the pros >> > >> >> >> > > > > > and >> > >> >> >> > > > > > > > cons of this property. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > - (plus) If you lose a TM for some reason, you >> can >> > >> still >> > >> >> recover even >> > >> >> >> > > > > > if >> > >> >> >> > > > > > > it >> > >> >> >> > > > > > > > doesn't come back. We still need to give it some >> time >> > >> to >> > >> >> eliminate >> > >> >> >> > > > > > > > unnecessary rescaling, which can be controlled by >> > >> setting >> > >> >> >> > > > > > > > "resource-stabilization-timeout." >> > >> >> >> > > > > > > > - (plus) The resources can arrive with a >> significant >> > >> >> delay. For >> > >> >> >> > > > > > example, >> > >> >> >> > > > > > > > you're unable to spawn enough TMs on time because >> > >> you've >> > >> >> run out of >> > >> >> >> > > > > > > > resources in your k8s cluster, and you need to >> wait >> > >> for >> > >> >> the cluster >> > >> >> >> > > > > > auto >> > >> >> >> > > > > > > > scaler to kick in and add new nodes to the >> cluster. In >> > >> >> this scenario, >> > >> >> >> > > > > > > > you'll be able to start making progress faster, >> at the >> > >> >> cost of >> > >> >> >> > > > > multiple >> > >> >> >> > > > > > > > rescalings (once the remaining resources arrive). >> > >> >> >> > > > > > > > - (plus) This plays well with the declarative >> manner >> > >> of >> > >> >> today's >> > >> >> >> > > > > > > > infrastructure. For example, you tell k8s that >> you >> > >> need >> > >> >> 10 TMs, and >> > >> >> >> > > > > > > you'll >> > >> >> >> > > > > > > > eventually get them. >> > >> >> >> > > > > > > > - (minus) In the case of large state jobs, the >> cost of >> > >> >> multiple >> > >> >> >> > > > > > > rescalings >> > >> >> >> > > > > > > > might outweigh the above. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > We've already touched on the solution to this >> problem >> > >> on >> > >> >> the FLIP. >> > >> >> >> > > > > > Please >> > >> >> >> > > > > > > > notice the parallelism knob being a range with a >> lower >> > >> >> and upper >> > >> >> >> > > > > bound. >> > >> >> >> > > > > > > > Setting both the lower and upper bound to the >> same >> > >> value >> > >> >> could give >> > >> >> >> > > > > the >> > >> >> >> > > > > > > > behavior you're describing at the cost of giving >> up >> > >> some >> > >> >> properties >> > >> >> >> > > > > > that >> > >> >> >> > > > > > > AS >> > >> >> >> > > > > > > > gives you (you'd be falling back to the >> > >> >> DefaultScheduler's behavior). >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > when user upgrade job-vertx-parallelism . I want >> to >> > >> have >> > >> >> an interface >> > >> >> >> > > > > > to >> > >> >> >> > > > > > > > > query the current update parallel execution >> status, >> > >> so >> > >> >> that the >> > >> >> >> > > > > user >> > >> >> >> > > > > > or >> > >> >> >> > > > > > > > > program can understand the current status >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > This is a misunderstanding. We're not >> introducing the >> > >> >> RESCALE >> > >> >> >> > > > > endpoint. >> > >> >> >> > > > > > > > This endpoint allows you to re-declare the >> resources >> > >> >> needed to run >> > >> >> >> > > > > the >> > >> >> >> > > > > > > job. >> > >> >> >> > > > > > > > Once you reach the desired resources (you get >> more >> > >> >> resources than the >> > >> >> >> > > > > > > lower >> > >> >> >> > > > > > > > bound defines), your job will run. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > We can expose a similar endpoint to "resource >> > >> >> requirements" to give >> > >> >> >> > > > > you >> > >> >> >> > > > > > > an >> > >> >> >> > > > > > > > overview of the resources the vertices already >> have. >> > >> You >> > >> >> can already >> > >> >> >> > > > > > get >> > >> >> >> > > > > > > > this from the REST API, so exposing this in yet >> > >> another >> > >> >> way should be >> > >> >> >> > > > > > > > considered carefully. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > *@Matthias* >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > I'm wondering whether it makes sense to add some >> kind >> > >> of >> > >> >> resource ID >> > >> >> >> > > > > to >> > >> >> >> > > > > > > the >> > >> >> >> > > > > > > > > REST API. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > That's a good question. I want to think about >> that and >> > >> >> get back to >> > >> >> >> > > > > the >> > >> >> >> > > > > > > > question later. My main struggle when thinking >> about >> > >> this >> > >> >> is, "if >> > >> >> >> > > > > this >> > >> >> >> > > > > > > > would be an idempotent POST endpoint," would it >> be any >> > >> >> different? >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > How often do we allow resource requirements to be >> > >> changed? >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > There shall be no rate limiting on the FLINK >> side. If >> > >> >> this is >> > >> >> >> > > > > something >> > >> >> >> > > > > > > > your environment needs, you can achieve it on a >> > >> different >> > >> >> layer ("we >> > >> >> >> > > > > > > can't >> > >> >> >> > > > > > > > have FLINK to do everything"). >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Versioning the JobGraph in the JobGraphStore >> rather >> > >> than >> > >> >> overwriting >> > >> >> >> > > > > it >> > >> >> >> > > > > > > > > might be an idea. >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > This sounds interesting since it would be closer >> to >> > >> the >> > >> >> JobGraph >> > >> >> >> > > > > being >> > >> >> >> > > > > > > > immutable. The main problem I see here is that >> this >> > >> would >> > >> >> introduce a >> > >> >> >> > > > > > > > BW-incompatible change so it might be a topic for >> > >> >> follow-up FLIP. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > I'm just wondering whether we bundle two things >> > >> together >> > >> >> that are >> > >> >> >> > > > > > > actually >> > >> >> >> > > > > > > > > separate >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Yup, this is how we think about it as well. The >> main >> > >> >> question is, >> > >> >> >> > > > > "who >> > >> >> >> > > > > > > > should be responsible for bookkeeping 1) the >> JobGraph >> > >> and >> > >> >> 2) the >> > >> >> >> > > > > > > > JobResourceRequirements". The JobMaster would be >> the >> > >> >> right place for >> > >> >> >> > > > > > > both, >> > >> >> >> > > > > > > > but it's currently not the case, and we're >> tightly >> > >> >> coupling the >> > >> >> >> > > > > > > dispatcher >> > >> >> >> > > > > > > > with the JobMaster. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Initially, we tried to introduce a separate HA >> > >> component >> > >> >> in JobMaster >> > >> >> >> > > > > > for >> > >> >> >> > > > > > > > bookkeeping the JobResourceRequirements, but that >> > >> proved >> > >> >> to be a more >> > >> >> >> > > > > > > > significant effort adding additional mess to the >> > >> already >> > >> >> messy HA >> > >> >> >> > > > > > > > ecosystem. Another approach we've discussed was >> > >> mutating >> > >> >> the JobGraph >> > >> >> >> > > > > > and >> > >> >> >> > > > > > > > setting JRR into the JobGraph structure itself. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > The middle ground for keeping this effort >> reasonably >> > >> >> sized and not >> > >> >> >> > > > > > > > violating "we want to keep JG immutable" too >> much is >> > >> >> keeping the >> > >> >> >> > > > > > > > JobResourceRequirements separate as an internal >> config >> > >> >> option in >> > >> >> >> > > > > > > JobGraph's >> > >> >> >> > > > > > > > configuration. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > We ultimately need to rethink the tight coupling >> of >> > >> >> Dispatcher and >> > >> >> >> > > > > > > > JobMaster, but it needs to be a separate effort. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > ...also considering the amount of data that can >> be >> > >> stored >> > >> >> in a >> > >> >> >> > > > > > > > > ConfigMap/ZooKeeper node if versioning the >> resource >> > >> >> requirement >> > >> >> >> > > > > > change >> > >> >> >> > > > > > > as >> > >> >> >> > > > > > > > > proposed in my previous item is an option for >> us. >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > AFAIK we're only storing pointers to the S3 >> objects >> > >> in HA >> > >> >> metadata, >> > >> >> >> > > > > so >> > >> >> >> > > > > > we >> > >> >> >> > > > > > > > should be okay with having larger structures for >> now. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Updating the JobGraphStore means adding more >> requests >> > >> to >> > >> >> the HA >> > >> >> >> > > > > backend >> > >> >> >> > > > > > > > API. >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > It's fine unless you intend to override the >> resource >> > >> >> requirements a >> > >> >> >> > > > > few >> > >> >> >> > > > > > > > times per second. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > *@Shammon* >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > How about adding some more information such as >> vertex >> > >> type >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Since it was intended as a "debug" endpoint, it >> makes >> > >> >> complete sense! >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > For sql jobs, we always use a unified >> parallelism for >> > >> >> most vertices. >> > >> >> >> > > > > > Can >> > >> >> >> > > > > > > > > we provide them with a more convenient setting >> > >> method >> > >> >> instead of >> > >> >> >> > > > > each >> > >> >> >> > > > > > > > one? >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > I completely feel with this. The main thoughts >> when >> > >> >> designing the API >> > >> >> >> > > > > > > were: >> > >> >> >> > > > > > > > - We want to keep it clean and easy to >> understand. >> > >> >> >> > > > > > > > - Global parallelism can be modeled using >> per-vertex >> > >> >> parallelism but >> > >> >> >> > > > > > not >> > >> >> >> > > > > > > > the other way around. >> > >> >> >> > > > > > > > - The API will be used by external tooling >> (operator, >> > >> >> auto scaler). >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Can you elaborate more about how you'd intend to >> use >> > >> the >> > >> >> endpoint? I >> > >> >> >> > > > > > > think >> > >> >> >> > > > > > > > we can ultimately introduce a way of re-declaring >> > >> >> "per-vertex >> > >> >> >> > > > > > defaults," >> > >> >> >> > > > > > > > but I'd like to understand the use case bit more >> > >> first. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > *@Weijie* >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > What is the default value here (based on what >> > >> >> configuration), or just >> > >> >> >> > > > > > > > > infinite? >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Currently, for the lower bound, it's always one, >> and >> > >> for >> > >> >> the upper >> > >> >> >> > > > > > bound, >> > >> >> >> > > > > > > > it's either parallelism (if defined) or the >> > >> >> maxParallelism of the >> > >> >> >> > > > > > vertex >> > >> >> >> > > > > > > in >> > >> >> >> > > > > > > > JobGraph. This question might be another signal >> for >> > >> >> making the >> > >> >> >> > > > > defaults >> > >> >> >> > > > > > > > explicit (see the answer to Shammon's question >> above). >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Thanks, everyone, for your initial thoughts! >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > Best, >> > >> >> >> > > > > > > > D. >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > On Tue, Feb 7, 2023 at 4:39 AM weijie guo < >> > >> >> guoweijieres...@gmail.com >> > >> >> >> > > > > > >> > >> >> >> > > > > > > > wrote: >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > > Thanks David for driving this. This is a very >> > >> valuable >> > >> >> work, >> > >> >> >> > > > > > especially >> > >> >> >> > > > > > > > for >> > >> >> >> > > > > > > > > cloud native environment. >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > >> How about adding some more information such >> as >> > >> >> vertex type >> > >> >> >> > > > > > > > > (SOURCE/MAP/JOIN and .etc) in the response of >> `get >> > >> jobs >> > >> >> >> > > > > > > > > resource-requirements`? For users, only >> vertex-id >> > >> may >> > >> >> be difficult >> > >> >> >> > > > > to >> > >> >> >> > > > > > > > > understand. >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > +1 for this suggestion, including jobvertex's >> name >> > >> in >> > >> >> the response >> > >> >> >> > > > > > body >> > >> >> >> > > > > > > > is >> > >> >> >> > > > > > > > > more >> > >> >> >> > > > > > > > > user-friendly. >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > I saw this sentence in FLIP: "Setting the upper >> > >> bound >> > >> >> to -1 will >> > >> >> >> > > > > > reset >> > >> >> >> > > > > > > > the >> > >> >> >> > > > > > > > > value to the default setting." What is the >> default >> > >> >> value here >> > >> >> >> > > > > (based >> > >> >> >> > > > > > > on >> > >> >> >> > > > > > > > > what configuration), or just infinite? >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > Best regards, >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > Weijie >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > Shammon FY <zjur...@gmail.com> 于2023年2月6日周一 >> > >> 18:06写道： >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > > > Hi David >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > Thanks for initiating this discussion. I >> think >> > >> >> declaring job >> > >> >> >> > > > > > resource >> > >> >> >> > > > > > > > > > requirements by REST API is very valuable. I >> just >> > >> >> left some >> > >> >> >> > > > > > comments >> > >> >> >> > > > > > > as >> > >> >> >> > > > > > > > > > followed >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > 1) How about adding some more information >> such as >> > >> >> vertex type >> > >> >> >> > > > > > > > > > (SOURCE/MAP/JOIN and .etc) in the response >> of `get >> > >> >> jobs >> > >> >> >> > > > > > > > > > resource-requirements`? For users, only >> vertex-id >> > >> may >> > >> >> be >> > >> >> >> > > > > difficult >> > >> >> >> > > > > > to >> > >> >> >> > > > > > > > > > understand. >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > 2) For sql jobs, we always use a unified >> > >> parallelism >> > >> >> for most >> > >> >> >> > > > > > > vertices. >> > >> >> >> > > > > > > > > Can >> > >> >> >> > > > > > > > > > we provide them with a more convenient >> setting >> > >> method >> > >> >> instead of >> > >> >> >> > > > > > each >> > >> >> >> > > > > > > > > one? >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > Best, >> > >> >> >> > > > > > > > > > Shammon >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > On Fri, Feb 3, 2023 at 8:18 PM Matthias Pohl >> < >> > >> >> >> > > > > > matthias.p...@aiven.io >> > >> >> >> > > > > > > > > > .invalid> >> > >> >> >> > > > > > > > > > wrote: >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > > > Thanks David for creating this FLIP. It >> sounds >> > >> >> promising and >> > >> >> >> > > > > > useful >> > >> >> >> > > > > > > > to >> > >> >> >> > > > > > > > > > > have. Here are some thoughts from my side >> (some >> > >> of >> > >> >> them might >> > >> >> >> > > > > be >> > >> >> >> > > > > > > > > rather a >> > >> >> >> > > > > > > > > > > follow-up and not necessarily part of this >> > >> FLIP): >> > >> >> >> > > > > > > > > > > - I'm wondering whether it makes sense to >> add >> > >> some >> > >> >> kind of >> > >> >> >> > > > > > resource >> > >> >> >> > > > > > > > ID >> > >> >> >> > > > > > > > > to >> > >> >> >> > > > > > > > > > > the REST API. This would give Flink a tool >> to >> > >> >> verify the PATCH >> > >> >> >> > > > > > > > request >> > >> >> >> > > > > > > > > of >> > >> >> >> > > > > > > > > > > the external system in a compare-and-set >> kind of >> > >> >> manner. AFAIU, >> > >> >> >> > > > > > the >> > >> >> >> > > > > > > > > > process >> > >> >> >> > > > > > > > > > > requires the external system to retrieve >> the >> > >> >> resource >> > >> >> >> > > > > > requirements >> > >> >> >> > > > > > > > > first >> > >> >> >> > > > > > > > > > > (to retrieve the vertex IDs). A resource ID >> > >> <ABC> >> > >> >> would be sent >> > >> >> >> > > > > > > along >> > >> >> >> > > > > > > > > as >> > >> >> >> > > > > > > > > > a >> > >> >> >> > > > > > > > > > > unique identifier for the provided setup. >> It's >> > >> >> essentially the >> > >> >> >> > > > > > > > version >> > >> >> >> > > > > > > > > ID >> > >> >> >> > > > > > > > > > > of the currently deployed resource >> requirement >> > >> >> configuration. >> > >> >> >> > > > > > Flink >> > >> >> >> > > > > > > > > > doesn't >> > >> >> >> > > > > > > > > > > know whether the external system would use >> the >> > >> >> provided >> > >> >> >> > > > > > information >> > >> >> >> > > > > > > > in >> > >> >> >> > > > > > > > > > some >> > >> >> >> > > > > > > > > > > way to derive a new set of resource >> requirements >> > >> >> for this job. >> > >> >> >> > > > > > The >> > >> >> >> > > > > > > > > > > subsequent PATCH request with updated >> resource >> > >> >> requirements >> > >> >> >> > > > > would >> > >> >> >> > > > > > > > > include >> > >> >> >> > > > > > > > > > > the previously retrieved resource ID >> <ABC>. The >> > >> >> PATCH call >> > >> >> >> > > > > would >> > >> >> >> > > > > > > fail >> > >> >> >> > > > > > > > > if >> > >> >> >> > > > > > > > > > > there was a concurrent PATCH call in >> between >> > >> >> indicating to the >> > >> >> >> > > > > > > > external >> > >> >> >> > > > > > > > > > > system that the resource requirements were >> > >> >> concurrently >> > >> >> >> > > > > updated. >> > >> >> >> > > > > > > > > > > - How often do we allow resource >> requirements >> > >> to be >> > >> >> changed? >> > >> >> >> > > > > That >> > >> >> >> > > > > > > > > > question >> > >> >> >> > > > > > > > > > > might make my previous comment on the >> resource >> > >> ID >> > >> >> obsolete >> > >> >> >> > > > > > because >> > >> >> >> > > > > > > we >> > >> >> >> > > > > > > > > > could >> > >> >> >> > > > > > > > > > > just make any PATCH call fail if there was >> a >> > >> >> resource >> > >> >> >> > > > > requirement >> > >> >> >> > > > > > > > > update >> > >> >> >> > > > > > > > > > > within a certain time frame before the >> request. >> > >> But >> > >> >> such a time >> > >> >> >> > > > > > > > period >> > >> >> >> > > > > > > > > is >> > >> >> >> > > > > > > > > > > something we might want to make >> configurable >> > >> then, >> > >> >> I guess. >> > >> >> >> > > > > > > > > > > - Versioning the JobGraph in the >> JobGraphStore >> > >> >> rather than >> > >> >> >> > > > > > > > overwriting >> > >> >> >> > > > > > > > > it >> > >> >> >> > > > > > > > > > > might be an idea. This would enable us to >> > >> provide >> > >> >> resource >> > >> >> >> > > > > > > > requirement >> > >> >> >> > > > > > > > > > > changes in the UI or through the REST API. >> It is >> > >> >> related to a >> > >> >> >> > > > > > > problem >> > >> >> >> > > > > > > > > > > around keeping track of the exception >> history >> > >> >> within the >> > >> >> >> > > > > > > > > > AdaptiveScheduler >> > >> >> >> > > > > > > > > > > and also having to consider multiple >> versions >> > >> of a >> > >> >> JobGraph. >> > >> >> >> > > > > But >> > >> >> >> > > > > > > for >> > >> >> >> > > > > > > > > that >> > >> >> >> > > > > > > > > > > one, we use the ExecutionGraphInfoStore >> right >> > >> now. >> > >> >> >> > > > > > > > > > > - Updating the JobGraph in the >> JobGraphStore >> > >> makes >> > >> >> sense. I'm >> > >> >> >> > > > > > just >> > >> >> >> > > > > > > > > > > wondering whether we bundle two things >> together >> > >> >> that are >> > >> >> >> > > > > actually >> > >> >> >> > > > > > > > > > separate: >> > >> >> >> > > > > > > > > > > The business logic and the execution >> > >> configuration >> > >> >> (the >> > >> >> >> > > > > resource >> > >> >> >> > > > > > > > > > > requirements). I'm aware that this is not a >> > >> flaw of >> > >> >> the current >> > >> >> >> > > > > > > FLIP >> > >> >> >> > > > > > > > > but >> > >> >> >> > > > > > > > > > > rather something that was not necessary to >> > >> address >> > >> >> in the past >> > >> >> >> > > > > > > > because >> > >> >> >> > > > > > > > > > the >> > >> >> >> > > > > > > > > > > JobGraph was kind of static. I don't >> remember >> > >> >> whether that was >> > >> >> >> > > > > > > > already >> > >> >> >> > > > > > > > > > > discussed while working on the >> AdaptiveScheduler >> > >> >> for FLIP-160 >> > >> >> >> > > > > > [1]. >> > >> >> >> > > > > > > > > Maybe, >> > >> >> >> > > > > > > > > > > I'm missing some functionality here that >> > >> requires >> > >> >> us to have >> > >> >> >> > > > > > > > everything >> > >> >> >> > > > > > > > > > in >> > >> >> >> > > > > > > > > > > one place. But it feels like updating the >> entire >> > >> >> JobGraph which >> > >> >> >> > > > > > > could >> > >> >> >> > > > > > > > > be >> > >> >> >> > > > > > > > > > > actually a "config change" is not >> reasonable. >> > >> >> ...also >> > >> >> >> > > > > considering >> > >> >> >> > > > > > > the >> > >> >> >> > > > > > > > > > > amount of data that can be stored in a >> > >> >> ConfigMap/ZooKeeper node >> > >> >> >> > > > > > if >> > >> >> >> > > > > > > > > > > versioning the resource requirement change >> as >> > >> >> proposed in my >> > >> >> >> > > > > > > previous >> > >> >> >> > > > > > > > > > item >> > >> >> >> > > > > > > > > > > is an option for us. >> > >> >> >> > > > > > > > > > > - Updating the JobGraphStore means adding >> more >> > >> >> requests to the >> > >> >> >> > > > > HA >> > >> >> >> > > > > > > > > backend >> > >> >> >> > > > > > > > > > > API. There were some concerns shared in the >> > >> >> discussion thread >> > >> >> >> > > > > [2] >> > >> >> >> > > > > > > for >> > >> >> >> > > > > > > > > > > FLIP-270 [3] on pressuring the k8s API >> server in >> > >> >> the past with >> > >> >> >> > > > > > too >> > >> >> >> > > > > > > > many >> > >> >> >> > > > > > > > > > > calls. Eventhough, it's more likely to be >> > >> caused by >> > >> >> >> > > > > > checkpointing, >> > >> >> >> > > > > > > I >> > >> >> >> > > > > > > > > > still >> > >> >> >> > > > > > > > > > > wanted to bring it up. We're working on a >> > >> >> standardized >> > >> >> >> > > > > > performance >> > >> >> >> > > > > > > > test >> > >> >> >> > > > > > > > > > to >> > >> >> >> > > > > > > > > > > prepare going forward with FLIP-270 [3] >> right >> > >> now. >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > > Best, >> > >> >> >> > > > > > > > > > > Matthias >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > > [1] >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > > >> > >> >> >> > > > > >> > >> >> >> > >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler >> > >> >> >> > > > > > > > > > > [2] >> > >> >> >> > > > > > > >> > >> >> https://lists.apache.org/thread/bm6rmxxk6fbrqfsgz71gvso58950d4mj >> > >> >> >> > > > > > > > > > > [3] >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > > >> > >> >> >> > > > > >> > >> >> >> > >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-270%3A+Repeatable+Cleanup+of+Checkpoints >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > > On Fri, Feb 3, 2023 at 10:31 AM ConradJam < >> > >> >> jam.gz...@gmail.com >> > >> >> >> > > > > > >> > >> >> >> > > > > > > > wrote: >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > Hi David: >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > Thank you for drive this flip, which >> helps >> > >> less >> > >> >> flink >> > >> >> >> > > > > shutdown >> > >> >> >> > > > > > > time >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > for this flip, I would like to make a few >> > >> idea on >> > >> >> share >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > - when the number of "slots" is >> > >> insufficient, >> > >> >> can we can >> > >> >> >> > > > > > stop >> > >> >> >> > > > > > > > > users >> > >> >> >> > > > > > > > > > > > rescaling or throw something to tell >> user >> > >> >> "less avaliable >> > >> >> >> > > > > > > slots >> > >> >> >> > > > > > > > to >> > >> >> >> > > > > > > > > > > > upgrade, >> > >> >> >> > > > > > > > > > > > please checkout your alivalbe slots" >> ? Or >> > >> we >> > >> >> could have a >> > >> >> >> > > > > > > > request >> > >> >> >> > > > > > > > > > > > switch(true/false) to allow this >> behavior >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > - when user upgrade >> job-vertx-parallelism >> > >> . I >> > >> >> want to have >> > >> >> >> > > > > > an >> > >> >> >> > > > > > > > > > > interface >> > >> >> >> > > > > > > > > > > > to query the current update parallel >> > >> execution >> > >> >> status, so >> > >> >> >> > > > > > that >> > >> >> >> > > > > > > > the >> > >> >> >> > > > > > > > > > > user >> > >> >> >> > > > > > > > > > > > or >> > >> >> >> > > > > > > > > > > > program can understand the current >> status >> > >> >> >> > > > > > > > > > > > - I want to have an interface to >> query the >> > >> >> current update >> > >> >> >> > > > > > > > > > parallelism >> > >> >> >> > > > > > > > > > > > execution status. This also helps >> similar >> > >> to >> > >> >> *[1] Flink >> > >> >> >> > > > > K8S >> > >> >> >> > > > > > > > > > Operator* >> > >> >> >> > > > > > > > > > > > management >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > { >> > >> >> >> > > > > > > > > > > > status: Failed >> > >> >> >> > > > > > > > > > > > reason: "less avaliable slots to >> upgrade, >> > >> >> please checkout >> > >> >> >> > > > > > your >> > >> >> >> > > > > > > > > > alivalbe >> > >> >> >> > > > > > > > > > > > slots" >> > >> >> >> > > > > > > > > > > > } >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > - *Pending*: this job now is join the >> > >> upgrade >> > >> >> queue,it >> > >> >> >> > > > > will >> > >> >> >> > > > > > be >> > >> >> >> > > > > > > > > > update >> > >> >> >> > > > > > > > > > > > later >> > >> >> >> > > > > > > > > > > > - *Rescaling*: job now is >> rescaling,wait it >> > >> >> finish >> > >> >> >> > > > > > > > > > > > - *Finished*: finish do it >> > >> >> >> > > > > > > > > > > > - *Failed* : something have wrong,so >> this >> > >> job >> > >> >> is not >> > >> >> >> > > > > > alivable >> > >> >> >> > > > > > > > > > upgrade >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > I want to supplement my above content in >> flip, >> > >> >> what do you >> > >> >> >> > > > > > think >> > >> >> >> > > > > > > ? >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > 1. >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > >> > >> >> >> > >> >> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > David Morávek <d...@apache.org> >> 于2023年2月3日周五 >> > >> >> 16:42写道： >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > Hi everyone, >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > This FLIP [1] introduces a new REST >> API for >> > >> >> declaring >> > >> >> >> > > > > > resource >> > >> >> >> > > > > > > > > > > > requirements >> > >> >> >> > > > > > > > > > > > > for the Adaptive Scheduler. There >> seems to >> > >> be a >> > >> >> clear need >> > >> >> >> > > > > > for >> > >> >> >> > > > > > > > this >> > >> >> >> > > > > > > > > > API >> > >> >> >> > > > > > > > > > > > > based on the discussion on the >> "Reworking >> > >> the >> > >> >> Rescale API" >> > >> >> >> > > > > > [2] >> > >> >> >> > > > > > > > > > thread. >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > Before we get started, this work is >> heavily >> > >> >> based on the >> > >> >> >> > > > > > > > prototype >> > >> >> >> > > > > > > > > > [3] >> > >> >> >> > > > > > > > > > > > > created by Till Rohrmann, and the FLIP >> is >> > >> being >> > >> >> published >> > >> >> >> > > > > > with >> > >> >> >> > > > > > > > his >> > >> >> >> > > > > > > > > > > > consent. >> > >> >> >> > > > > > > > > > > > > Big shoutout to him! >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > Last and not least, thanks to Chesnay >> and >> > >> Roman >> > >> >> for the >> > >> >> >> > > > > > initial >> > >> >> >> > > > > > > > > > reviews >> > >> >> >> > > > > > > > > > > > and >> > >> >> >> > > > > > > > > > > > > discussions. >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > The best start would be watching a >> short >> > >> demo >> > >> >> [4] that I've >> > >> >> >> > > > > > > > > recorded, >> > >> >> >> > > > > > > > > > > > which >> > >> >> >> > > > > > > > > > > > > illustrates newly added capabilities >> > >> (rescaling >> > >> >> the running >> > >> >> >> > > > > > > job, >> > >> >> >> > > > > > > > > > > handing >> > >> >> >> > > > > > > > > > > > > back resources to the RM, and session >> > >> cluster >> > >> >> support). >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > The intuition behind the FLIP is being >> able >> > >> to >> > >> >> define >> > >> >> >> > > > > > resource >> > >> >> >> > > > > > > > > > > > requirements >> > >> >> >> > > > > > > > > > > > > ("resource boundaries") externally >> that the >> > >> >> >> > > > > AdaptiveScheduler >> > >> >> >> > > > > > > can >> > >> >> >> > > > > > > > > > > > navigate >> > >> >> >> > > > > > > > > > > > > within. This is a building block for >> > >> >> higher-level efforts >> > >> >> >> > > > > > such >> > >> >> >> > > > > > > as >> > >> >> >> > > > > > > > > an >> > >> >> >> > > > > > > > > > > > > external Autoscaler. The natural >> extension >> > >> of >> > >> >> this work >> > >> >> >> > > > > would >> > >> >> >> > > > > > > be >> > >> >> >> > > > > > > > to >> > >> >> >> > > > > > > > > > > allow >> > >> >> >> > > > > > > > > > > > > to specify per-vertex ResourceProfiles. >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > Looking forward to your thoughts; any >> > >> feedback >> > >> >> is >> > >> >> >> > > > > > appreciated! >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > [1] >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > > >> > >> >> >> > > > > >> > >> >> >> > >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management >> > >> >> >> > > > > > > > > > > > > [2] >> > >> >> >> > > > > > > > > >> > >> >> https://lists.apache.org/thread/2f7dgr88xtbmsohtr0f6wmsvw8sw04f5 >> > >> >> >> > > > > > > > > > > > > [3] >> > >> >> https://github.com/tillrohrmann/flink/tree/autoscaling >> > >> >> >> > > > > > > > > > > > > [4] >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > >> > >> >> >> https://drive.google.com/file/d/1Vp8W-7Zk_iKXPTAiBT-eLPmCMd_I57Ty/view >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > > Best, >> > >> >> >> > > > > > > > > > > > > D. >> > >> >> >> > > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > -- >> > >> >> >> > > > > > > > > > > > Best >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > > ConradJam >> > >> >> >> > > > > > > > > > > > >> > >> >> >> > > > > > > > > > > >> > >> >> >> > > > > > > > > > >> > >> >> >> > > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > > > >> > >> >> >> > > > > >> > >> >> >> > >> >> > > >> >
