I missed that the FLIP states:

> Currently, even though we’d expose the lower bound for clarity and API 
> completeness, we won’t allow setting it to any other value than one until we 
> have full support throughout the stack.

How hard would it be to address this issue in the FLIP?

There is not much value to offer setting a lower bound which won't be
respected / throw an error when it is set. If we had support for a
lower bound, we could enforce a resource contract externally via
setting lowerBound == upperBound. That ties back to the Rescale API
discussion we had. I want to better understand what the major concerns
would be around allowing this.

Just to outline how I imagine the logic to work:

A) The resource constraints are already met => Nothing changes
B) More resources available than required => Cancel the job, drop as
many superfluous task managers as needed, restart the job
C) Less resources available than required => Acquire new task
managers, wait for them to register, cancel and restart the job

I'm open to helping out with the implementation.

-Max

On Mon, Feb 13, 2023 at 7:45 PM Maximilian Michels <m...@apache.org> wrote:
>
> Based on further discussion I had with Chesnay on this PR [1], I think
> jobs would currently go into a restarting state after the resource
> requirements have changed. This wouldn't achieve what we had in mind,
> i.e. sticking to the old resource requirements until enough slots are
> available to fulfil the new resource requirements. So this may not be
> 100% what we need but it could be extended to do what we want.
>
> -Max
>
> [1] https://github.com/apache/flink/pull/21908#discussion_r1104792362
>
> On Mon, Feb 13, 2023 at 7:16 PM Maximilian Michels <m...@apache.org> wrote:
> >
> > Hi David,
> >
> > This is awesome! Great writeup and demo. This is pretty much what we
> > need for the autoscaler as part of the Flink Kubernetes operator [1].
> > Scaling Flink jobs effectively is hard but fortunately we have solved
> > the issue as part of the Flink Kubernetes operator. The only critical
> > piece we are missing is a better way to execute scaling decisions, as
> > discussed in [2].
> >
> > Looking at your proposal, we would set lowerBound == upperBound for
> > the parallelism because we want to fully determine the parallelism
> > externally based on the scaling metrics. Does that sound right?
> >
> > What is the timeline for these changes? Is there a JIRA?
> >
> > Cheers,
> > Max
> >
> > [1] 
> > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/
> > [2] https://lists.apache.org/thread/2f7dgr88xtbmsohtr0f6wmsvw8sw04f5
> >
> > On Mon, Feb 13, 2023 at 1:16 PM feng xiangyu <xiangyu...@gmail.com> wrote:
> > >
> > > Hi David,
> > >
> > > Thanks for your reply.  I think your response totally make sense.  This
> > > flip targets on declaring required resource to ResourceManager instead of
> > > using  ResourceManager to add/remove TMs directly.
> > >
> > > Best,
> > > Xiangyu
> > >
> > >
> > >
> > > David Morávek <david.mora...@gmail.com> 于2023年2月13日周一 15:46写道:
> > >
> > > > Hi everyone,
> > > >
> > > > @Shammon
> > > >
> > > > I'm not entirely sure what "config file" you're referring to. You can, 
> > > > of
> > > > course, override the default parallelism in "flink-conf.yaml", but for
> > > > sinks and sources, the parallelism needs to be tweaked on the connector
> > > > level ("WITH" statement).
> > > >
> > > > This is something that should be achieved with tooling around Flink. We
> > > > want to provide an API on the lowest level that generalizes well. 
> > > > Achieving
> > > > what you're describing should be straightforward with this API.
> > > >
> > > > @Xiangyu
> > > >
> > > > Is it possible for this REST API to declare TM resources in the future?
> > > >
> > > >
> > > > Would you like to add/remove TMs if you use an active Resource Manager?
> > > > This would be out of the scope of this effort since it targets the
> > > > scheduler component only (we make no assumptions about the used Resource
> > > > Manager). Also, the AdaptiveScheduler is only intended to be used for
> > > > Streaming.
> > > >
> > > >  And for streaming jobs, I'm wondering if there is any situation we 
> > > > need to
> > > > > rescale the TM resources of a flink cluster at first and then the
> > > > adaptive
> > > > > scheduler will rescale the per-vertex ResourceProfiles accordingly.
> > > > >
> > > >
> > > > We plan on adding support for the ResourceProfiles (dynamic slot
> > > > allocation) as the next step. Again we won't make any assumptions about 
> > > > the
> > > > used Resource Manager. In other words, this effort ends by declaring
> > > > desired resources to the Resource Manager.
> > > >
> > > > Does that make sense?
> > > >
> > > > @Matthias
> > > >
> > > > We've done another pass on the proposed API and currently lean towards
> > > > having an idempotent PUT API.
> > > > - We don't care too much about multiple writers' scenarios in terms of 
> > > > who
> > > > can write an authoritative payload; this is up to the user of the API to
> > > > figure out
> > > > - It's indeed tricky to achieve atomicity with PATCH API; switching to 
> > > > PUT
> > > > API seems to do the trick
> > > > - We won't allow partial "payloads" anymore, meaning you need to define
> > > > requirements for all vertices in the JobGraph; This is completely fine 
> > > > for
> > > > the programmatic workflows. For DEBUG / DEMO purposes, you can use the 
> > > > GET
> > > > endpoint and tweak the response to avoid writing the whole payload by 
> > > > hand.
> > > >
> > > > WDYT?
> > > >
> > > >
> > > > Best,
> > > > D.
> > > >
> > > > On Fri, Feb 10, 2023 at 11:21 AM feng xiangyu <xiangyu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > Thanks for creating this flip. I think this work it is very useful,
> > > > > especially in autoscaling scenario.  I would like to share some 
> > > > > questions
> > > > > from my view.
> > > > >
> > > > > 1, Is it possible for this REST API to declare TM resources in the
> > > > future?
> > > > > I'm asking because we are building the autoscaling feature for Flink 
> > > > > OLAP
> > > > > Session Cluster in ByteDance. We need to rescale the cluster's 
> > > > > resource
> > > > on
> > > > > TM level instead of Job level. It would be very helpful if we have a 
> > > > > REST
> > > > > API for out external Autoscaling service to use.
> > > > >
> > > > > 2, And for streaming jobs, I'm wondering if there is any situation we
> > > > need
> > > > > to rescale the TM resources of a flink cluster at first and then the
> > > > > adaptive scheduler will rescale the per-vertex ResourceProfiles
> > > > > accordingly.
> > > > >
> > > > > best.
> > > > > Xiangyu
> > > > >
> > > > > Shammon FY <zjur...@gmail.com> 于2023年2月9日周四 11:31写道:
> > > > >
> > > > > > Hi David
> > > > > >
> > > > > > Thanks for your answer.
> > > > > >
> > > > > > > Can you elaborate more about how you'd intend to use the 
> > > > > > > endpoint? I
> > > > > > think we can ultimately introduce a way of re-declaring "per-vertex
> > > > > > defaults," but I'd like to understand the use case bit more first.
> > > > > >
> > > > > > For this issue, I mainly consider the consistency of user 
> > > > > > configuration
> > > > > and
> > > > > > job runtime. For sql jobs, users usually set specific parallelism 
> > > > > > for
> > > > > > source and sink, and set a global parallelism for other operators.
> > > > These
> > > > > > config items are stored in a config file. For some high-priority 
> > > > > > jobs,
> > > > > > users may want to manage them manually.
> > > > > > 1. When users need to scale the parallelism, they should update the
> > > > > config
> > > > > > file and restart flink job, which may take a long time.
> > > > > > 2. After providing the REST API, users can just send a request to 
> > > > > > the
> > > > job
> > > > > > via REST API quickly after updating the config file.
> > > > > > The configuration in the running job and config file should be the
> > > > same.
> > > > > > What do you think of this?
> > > > > >
> > > > > > best.
> > > > > > Shammon
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 7, 2023 at 4:51 PM David Morávek 
> > > > > > <david.mora...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Let's try to answer the questions one by one.
> > > > > > >
> > > > > > > *@ConradJam*
> > > > > > >
> > > > > > > when the number of "slots" is insufficient, can we can stop users
> > > > > > rescaling
> > > > > > > > or throw something to tell user "less avaliable slots to 
> > > > > > > > upgrade,
> > > > > > please
> > > > > > > > checkout your alivalbe slots" ?
> > > > > > > >
> > > > > > >
> > > > > > > The main property of AdaptiveScheduler is that it can adapt to
> > > > > "available
> > > > > > > resources," which means you're still able to make progress even
> > > > though
> > > > > > you
> > > > > > > didn't get all the slots you've asked for. Let's break down the 
> > > > > > > pros
> > > > > and
> > > > > > > cons of this property.
> > > > > > >
> > > > > > > - (plus) If you lose a TM for some reason, you can still recover 
> > > > > > > even
> > > > > if
> > > > > > it
> > > > > > > doesn't come back. We still need to give it some time to eliminate
> > > > > > > unnecessary rescaling, which can be controlled by setting
> > > > > > > "resource-stabilization-timeout."
> > > > > > > - (plus) The resources can arrive with a significant delay. For
> > > > > example,
> > > > > > > you're unable to spawn enough TMs on time because you've run out 
> > > > > > > of
> > > > > > > resources in your k8s cluster, and you need to wait for the 
> > > > > > > cluster
> > > > > auto
> > > > > > > scaler to kick in and add new nodes to the cluster. In this 
> > > > > > > scenario,
> > > > > > > you'll be able to start making progress faster, at the cost of
> > > > multiple
> > > > > > > rescalings (once the remaining resources arrive).
> > > > > > > - (plus) This plays well with the declarative manner of today's
> > > > > > > infrastructure. For example, you tell k8s that you need 10 TMs, 
> > > > > > > and
> > > > > > you'll
> > > > > > > eventually get them.
> > > > > > > - (minus) In the case of large state jobs, the cost of multiple
> > > > > > rescalings
> > > > > > > might outweigh the above.
> > > > > > >
> > > > > > > We've already touched on the solution to this problem on the FLIP.
> > > > > Please
> > > > > > > notice the parallelism knob being a range with a lower and upper
> > > > bound.
> > > > > > > Setting both the lower and upper bound to the same value could 
> > > > > > > give
> > > > the
> > > > > > > behavior you're describing at the cost of giving up some 
> > > > > > > properties
> > > > > that
> > > > > > AS
> > > > > > > gives you (you'd be falling back to the DefaultScheduler's 
> > > > > > > behavior).
> > > > > > >
> > > > > > > when user upgrade job-vertx-parallelism . I want to have an 
> > > > > > > interface
> > > > > to
> > > > > > > > query the current update parallel execution status, so that the
> > > > user
> > > > > or
> > > > > > > > program can understand the current status
> > > > > > > >
> > > > > > >
> > > > > > > This is a misunderstanding. We're not introducing the RESCALE
> > > > endpoint.
> > > > > > > This endpoint allows you to re-declare the resources needed to run
> > > > the
> > > > > > job.
> > > > > > > Once you reach the desired resources (you get more resources than 
> > > > > > > the
> > > > > > lower
> > > > > > > bound defines), your job will run.
> > > > > > >
> > > > > > > We can expose a similar endpoint to "resource requirements" to 
> > > > > > > give
> > > > you
> > > > > > an
> > > > > > > overview of the resources the vertices already have. You can 
> > > > > > > already
> > > > > get
> > > > > > > this from the REST API, so exposing this in yet another way 
> > > > > > > should be
> > > > > > > considered carefully.
> > > > > > >
> > > > > > > *@Matthias*
> > > > > > >
> > > > > > > I'm wondering whether it makes sense to add some kind of resource 
> > > > > > > ID
> > > > to
> > > > > > the
> > > > > > > > REST API.
> > > > > > >
> > > > > > >
> > > > > > > That's a good question. I want to think about that and get back to
> > > > the
> > > > > > > question later. My main struggle when thinking about this is, "if
> > > > this
> > > > > > > would be an idempotent POST endpoint," would it be any different?
> > > > > > >
> > > > > > > How often do we allow resource requirements to be changed?
> > > > > > >
> > > > > > >
> > > > > > > There shall be no rate limiting on the FLINK side. If this is
> > > > something
> > > > > > > your environment needs, you can achieve it on a different layer 
> > > > > > > ("we
> > > > > > can't
> > > > > > > have FLINK to do everything").
> > > > > > >
> > > > > > > Versioning the JobGraph in the JobGraphStore rather than 
> > > > > > > overwriting
> > > > it
> > > > > > > > might be an idea.
> > > > > > > >
> > > > > > >
> > > > > > > This sounds interesting since it would be closer to the JobGraph
> > > > being
> > > > > > > immutable. The main problem I see here is that this would 
> > > > > > > introduce a
> > > > > > > BW-incompatible change so it might be a topic for follow-up FLIP.
> > > > > > >
> > > > > > > I'm just wondering whether we bundle two things together that are
> > > > > > actually
> > > > > > > > separate
> > > > > > > >
> > > > > > >
> > > > > > > Yup, this is how we think about it as well. The main question is,
> > > > "who
> > > > > > > should be responsible for bookkeeping 1) the JobGraph and 2) the
> > > > > > > JobResourceRequirements". The JobMaster would be the right place 
> > > > > > > for
> > > > > > both,
> > > > > > > but it's currently not the case, and we're tightly coupling the
> > > > > > dispatcher
> > > > > > > with the JobMaster.
> > > > > > >
> > > > > > > Initially, we tried to introduce a separate HA component in 
> > > > > > > JobMaster
> > > > > for
> > > > > > > bookkeeping the JobResourceRequirements, but that proved to be a 
> > > > > > > more
> > > > > > > significant effort adding additional mess to the already messy HA
> > > > > > > ecosystem. Another approach we've discussed was mutating the 
> > > > > > > JobGraph
> > > > > and
> > > > > > > setting JRR into the JobGraph structure itself.
> > > > > > >
> > > > > > > The middle ground for keeping this effort reasonably sized and not
> > > > > > > violating "we want to keep JG immutable" too much is keeping the
> > > > > > > JobResourceRequirements separate as an internal config option in
> > > > > > JobGraph's
> > > > > > > configuration.
> > > > > > >
> > > > > > > We ultimately need to rethink the tight coupling of Dispatcher and
> > > > > > > JobMaster, but it needs to be a separate effort.
> > > > > > >
> > > > > > > ...also considering the amount of data that can be stored in a
> > > > > > > > ConfigMap/ZooKeeper node if versioning the resource requirement
> > > > > change
> > > > > > as
> > > > > > > > proposed in my previous item is an option for us.
> > > > > > > >
> > > > > > >
> > > > > > > AFAIK we're only storing pointers to the S3 objects in HA 
> > > > > > > metadata,
> > > > so
> > > > > we
> > > > > > > should be okay with having larger structures for now.
> > > > > > >
> > > > > > > Updating the JobGraphStore means adding more requests to the HA
> > > > backend
> > > > > > > API.
> > > > > > > >
> > > > > > >
> > > > > > > It's fine unless you intend to override the resource requirements 
> > > > > > > a
> > > > few
> > > > > > > times per second.
> > > > > > >
> > > > > > > *@Shammon*
> > > > > > >
> > > > > > > How about adding some more information such as vertex type
> > > > > > > >
> > > > > > >
> > > > > > > Since it was intended as a "debug" endpoint, it makes complete 
> > > > > > > sense!
> > > > > > >
> > > > > > >  For sql jobs, we always use a unified parallelism for most 
> > > > > > > vertices.
> > > > > Can
> > > > > > > > we provide them with a more convenient setting method instead of
> > > > each
> > > > > > > one?
> > > > > > >
> > > > > > >
> > > > > > > I completely feel with this. The main thoughts when designing the 
> > > > > > > API
> > > > > > were:
> > > > > > > - We want to keep it clean and easy to understand.
> > > > > > > - Global parallelism can be modeled using per-vertex parallelism 
> > > > > > > but
> > > > > not
> > > > > > > the other way around.
> > > > > > > - The API will be used by external tooling (operator, auto 
> > > > > > > scaler).
> > > > > > >
> > > > > > > Can you elaborate more about how you'd intend to use the 
> > > > > > > endpoint? I
> > > > > > think
> > > > > > > we can ultimately introduce a way of re-declaring "per-vertex
> > > > > defaults,"
> > > > > > > but I'd like to understand the use case bit more first.
> > > > > > >
> > > > > > > *@Weijie*
> > > > > > >
> > > > > > > What is the default value here (based on what configuration), or 
> > > > > > > just
> > > > > > > > infinite?
> > > > > > > >
> > > > > > >
> > > > > > > Currently, for the lower bound, it's always one, and for the upper
> > > > > bound,
> > > > > > > it's either parallelism (if defined) or the maxParallelism of the
> > > > > vertex
> > > > > > in
> > > > > > > JobGraph. This question might be another signal for making the
> > > > defaults
> > > > > > > explicit (see the answer to Shammon's question above).
> > > > > > >
> > > > > > >
> > > > > > > Thanks, everyone, for your initial thoughts!
> > > > > > >
> > > > > > > Best,
> > > > > > > D.
> > > > > > >
> > > > > > > On Tue, Feb 7, 2023 at 4:39 AM weijie guo 
> > > > > > > <guoweijieres...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks David for driving this. This is a very valuable work,
> > > > > especially
> > > > > > > for
> > > > > > > > cloud native environment.
> > > > > > > >
> > > > > > > > >> How about adding some more information such as vertex type
> > > > > > > > (SOURCE/MAP/JOIN and .etc) in the response of `get jobs
> > > > > > > > resource-requirements`? For users, only vertex-id may be 
> > > > > > > > difficult
> > > > to
> > > > > > > > understand.
> > > > > > > >
> > > > > > > > +1 for this suggestion, including jobvertex's name in the 
> > > > > > > > response
> > > > > body
> > > > > > > is
> > > > > > > > more
> > > > > > > > user-friendly.
> > > > > > > >
> > > > > > > >
> > > > > > > > I saw this sentence in FLIP: "Setting the upper bound to -1 will
> > > > > reset
> > > > > > > the
> > > > > > > > value to the default setting."  What is the default value here
> > > > (based
> > > > > > on
> > > > > > > > what configuration), or just infinite?
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Weijie
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Shammon FY <zjur...@gmail.com> 于2023年2月6日周一 18:06写道:
> > > > > > > >
> > > > > > > > > Hi David
> > > > > > > > >
> > > > > > > > > Thanks for initiating this discussion. I think declaring job
> > > > > resource
> > > > > > > > > requirements by REST API is very valuable. I just left some
> > > > > comments
> > > > > > as
> > > > > > > > > followed
> > > > > > > > >
> > > > > > > > > 1) How about adding some more information such as vertex type
> > > > > > > > > (SOURCE/MAP/JOIN and .etc) in the response of `get jobs
> > > > > > > > > resource-requirements`? For users, only vertex-id may be
> > > > difficult
> > > > > to
> > > > > > > > > understand.
> > > > > > > > >
> > > > > > > > > 2) For sql jobs, we always use a unified parallelism for most
> > > > > > vertices.
> > > > > > > > Can
> > > > > > > > > we provide them with a more convenient setting method instead 
> > > > > > > > > of
> > > > > each
> > > > > > > > one?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 3, 2023 at 8:18 PM Matthias Pohl <
> > > > > matthias.p...@aiven.io
> > > > > > > > > .invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks David for creating this FLIP. It sounds promising and
> > > > > useful
> > > > > > > to
> > > > > > > > > > have. Here are some thoughts from my side (some of them 
> > > > > > > > > > might
> > > > be
> > > > > > > > rather a
> > > > > > > > > > follow-up and not necessarily part of this FLIP):
> > > > > > > > > > - I'm wondering whether it makes sense to add some kind of
> > > > > resource
> > > > > > > ID
> > > > > > > > to
> > > > > > > > > > the REST API. This would give Flink a tool to verify the 
> > > > > > > > > > PATCH
> > > > > > > request
> > > > > > > > of
> > > > > > > > > > the external system in a compare-and-set kind of manner. 
> > > > > > > > > > AFAIU,
> > > > > the
> > > > > > > > > process
> > > > > > > > > > requires the external system to retrieve the resource
> > > > > requirements
> > > > > > > > first
> > > > > > > > > > (to retrieve the vertex IDs). A resource ID <ABC> would be 
> > > > > > > > > > sent
> > > > > > along
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > unique identifier for the provided setup. It's essentially 
> > > > > > > > > > the
> > > > > > > version
> > > > > > > > ID
> > > > > > > > > > of the currently deployed resource requirement 
> > > > > > > > > > configuration.
> > > > > Flink
> > > > > > > > > doesn't
> > > > > > > > > > know whether the external system would use the provided
> > > > > information
> > > > > > > in
> > > > > > > > > some
> > > > > > > > > > way to derive a new set of resource requirements for this 
> > > > > > > > > > job.
> > > > > The
> > > > > > > > > > subsequent PATCH request with updated resource requirements
> > > > would
> > > > > > > > include
> > > > > > > > > > the previously retrieved resource ID <ABC>. The PATCH call
> > > > would
> > > > > > fail
> > > > > > > > if
> > > > > > > > > > there was a concurrent PATCH call in between indicating to 
> > > > > > > > > > the
> > > > > > > external
> > > > > > > > > > system that the resource requirements were concurrently
> > > > updated.
> > > > > > > > > > - How often do we allow resource requirements to be changed?
> > > > That
> > > > > > > > > question
> > > > > > > > > > might make my previous comment on the resource ID obsolete
> > > > > because
> > > > > > we
> > > > > > > > > could
> > > > > > > > > > just make any PATCH call fail if there was a resource
> > > > requirement
> > > > > > > > update
> > > > > > > > > > within a certain time frame before the request. But such a 
> > > > > > > > > > time
> > > > > > > period
> > > > > > > > is
> > > > > > > > > > something we might want to make configurable then, I guess.
> > > > > > > > > > - Versioning the JobGraph in the JobGraphStore rather than
> > > > > > > overwriting
> > > > > > > > it
> > > > > > > > > > might be an idea. This would enable us to provide resource
> > > > > > > requirement
> > > > > > > > > > changes in the UI or through the REST API. It is related to 
> > > > > > > > > > a
> > > > > > problem
> > > > > > > > > > around keeping track of the exception history within the
> > > > > > > > > AdaptiveScheduler
> > > > > > > > > > and also having to consider multiple versions of a JobGraph.
> > > > But
> > > > > > for
> > > > > > > > that
> > > > > > > > > > one, we use the ExecutionGraphInfoStore right now.
> > > > > > > > > > - Updating the JobGraph in the JobGraphStore makes sense. 
> > > > > > > > > > I'm
> > > > > just
> > > > > > > > > > wondering whether we bundle two things together that are
> > > > actually
> > > > > > > > > separate:
> > > > > > > > > > The business logic and the execution configuration (the
> > > > resource
> > > > > > > > > > requirements). I'm aware that this is not a flaw of the 
> > > > > > > > > > current
> > > > > > FLIP
> > > > > > > > but
> > > > > > > > > > rather something that was not necessary to address in the 
> > > > > > > > > > past
> > > > > > > because
> > > > > > > > > the
> > > > > > > > > > JobGraph was kind of static. I don't remember whether that 
> > > > > > > > > > was
> > > > > > > already
> > > > > > > > > > discussed while working on the AdaptiveScheduler for 
> > > > > > > > > > FLIP-160
> > > > > [1].
> > > > > > > > Maybe,
> > > > > > > > > > I'm missing some functionality here that requires us to have
> > > > > > > everything
> > > > > > > > > in
> > > > > > > > > > one place. But it feels like updating the entire JobGraph 
> > > > > > > > > > which
> > > > > > could
> > > > > > > > be
> > > > > > > > > > actually a "config change" is not reasonable. ...also
> > > > considering
> > > > > > the
> > > > > > > > > > amount of data that can be stored in a ConfigMap/ZooKeeper 
> > > > > > > > > > node
> > > > > if
> > > > > > > > > > versioning the resource requirement change as proposed in my
> > > > > > previous
> > > > > > > > > item
> > > > > > > > > > is an option for us.
> > > > > > > > > > - Updating the JobGraphStore means adding more requests to 
> > > > > > > > > > the
> > > > HA
> > > > > > > > backend
> > > > > > > > > > API. There were some concerns shared in the discussion 
> > > > > > > > > > thread
> > > > [2]
> > > > > > for
> > > > > > > > > > FLIP-270 [3] on pressuring the k8s API server in the past 
> > > > > > > > > > with
> > > > > too
> > > > > > > many
> > > > > > > > > > calls. Eventhough, it's more likely to be caused by
> > > > > checkpointing,
> > > > > > I
> > > > > > > > > still
> > > > > > > > > > wanted to bring it up. We're working on a standardized
> > > > > performance
> > > > > > > test
> > > > > > > > > to
> > > > > > > > > > prepare going forward with FLIP-270 [3] right now.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Matthias
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler
> > > > > > > > > > [2]
> > > > > > https://lists.apache.org/thread/bm6rmxxk6fbrqfsgz71gvso58950d4mj
> > > > > > > > > > [3]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-270%3A+Repeatable+Cleanup+of+Checkpoints
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 3, 2023 at 10:31 AM ConradJam 
> > > > > > > > > > <jam.gz...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi David:
> > > > > > > > > > >
> > > > > > > > > > > Thank you for drive this flip, which helps less flink
> > > > shutdown
> > > > > > time
> > > > > > > > > > >
> > > > > > > > > > > for this flip, I would like to make a few idea on share
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >    - when the number of "slots" is insufficient, can we 
> > > > > > > > > > > can
> > > > > stop
> > > > > > > > users
> > > > > > > > > > >    rescaling or throw something to tell user "less 
> > > > > > > > > > > avaliable
> > > > > > slots
> > > > > > > to
> > > > > > > > > > > upgrade,
> > > > > > > > > > >    please checkout your alivalbe slots" ? Or we could 
> > > > > > > > > > > have a
> > > > > > > request
> > > > > > > > > > >    switch(true/false) to allow this behavior
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >    - when user upgrade job-vertx-parallelism . I want to 
> > > > > > > > > > > have
> > > > > an
> > > > > > > > > > interface
> > > > > > > > > > >    to query the current update parallel execution status, 
> > > > > > > > > > > so
> > > > > that
> > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > or
> > > > > > > > > > >    program can understand the current status
> > > > > > > > > > >    - I want to have an interface to query the current 
> > > > > > > > > > > update
> > > > > > > > > parallelism
> > > > > > > > > > >    execution status. This also helps similar to *[1] Flink
> > > > K8S
> > > > > > > > > Operator*
> > > > > > > > > > >    management
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > {
> > > > > > > > > > >   status: Failed
> > > > > > > > > > >   reason: "less avaliable slots to upgrade, please 
> > > > > > > > > > > checkout
> > > > > your
> > > > > > > > > alivalbe
> > > > > > > > > > > slots"
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >    - *Pending*: this job now is join the upgrade queue,it
> > > > will
> > > > > be
> > > > > > > > > update
> > > > > > > > > > >    later
> > > > > > > > > > >    - *Rescaling*: job now is rescaling,wait it finish
> > > > > > > > > > >    - *Finished*: finish do it
> > > > > > > > > > >    - *Failed* : something have wrong,so this job is not
> > > > > alivable
> > > > > > > > > upgrade
> > > > > > > > > > >
> > > > > > > > > > > I want to supplement my above content in flip, what do you
> > > > > think
> > > > > > ?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >    1.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > David Morávek <d...@apache.org> 于2023年2月3日周五 16:42写道:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > This FLIP [1] introduces a new REST API for declaring
> > > > > resource
> > > > > > > > > > > requirements
> > > > > > > > > > > > for the Adaptive Scheduler. There seems to be a clear 
> > > > > > > > > > > > need
> > > > > for
> > > > > > > this
> > > > > > > > > API
> > > > > > > > > > > > based on the discussion on the "Reworking the Rescale 
> > > > > > > > > > > > API"
> > > > > [2]
> > > > > > > > > thread.
> > > > > > > > > > > >
> > > > > > > > > > > > Before we get started, this work is heavily based on the
> > > > > > > prototype
> > > > > > > > > [3]
> > > > > > > > > > > > created by Till Rohrmann, and the FLIP is being 
> > > > > > > > > > > > published
> > > > > with
> > > > > > > his
> > > > > > > > > > > consent.
> > > > > > > > > > > > Big shoutout to him!
> > > > > > > > > > > >
> > > > > > > > > > > > Last and not least, thanks to Chesnay and Roman for the
> > > > > initial
> > > > > > > > > reviews
> > > > > > > > > > > and
> > > > > > > > > > > > discussions.
> > > > > > > > > > > >
> > > > > > > > > > > > The best start would be watching a short demo [4] that 
> > > > > > > > > > > > I've
> > > > > > > > recorded,
> > > > > > > > > > > which
> > > > > > > > > > > > illustrates newly added capabilities (rescaling the 
> > > > > > > > > > > > running
> > > > > > job,
> > > > > > > > > > handing
> > > > > > > > > > > > back resources to the RM, and session cluster support).
> > > > > > > > > > > >
> > > > > > > > > > > > The intuition behind the FLIP is being able to define
> > > > > resource
> > > > > > > > > > > requirements
> > > > > > > > > > > > ("resource boundaries") externally that the
> > > > AdaptiveScheduler
> > > > > > can
> > > > > > > > > > > navigate
> > > > > > > > > > > > within. This is a building block for higher-level 
> > > > > > > > > > > > efforts
> > > > > such
> > > > > > as
> > > > > > > > an
> > > > > > > > > > > > external Autoscaler. The natural extension of this work
> > > > would
> > > > > > be
> > > > > > > to
> > > > > > > > > > allow
> > > > > > > > > > > > to specify per-vertex ResourceProfiles.
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to your thoughts; any feedback is
> > > > > appreciated!
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management
> > > > > > > > > > > > [2]
> > > > > > > > https://lists.apache.org/thread/2f7dgr88xtbmsohtr0f6wmsvw8sw04f5
> > > > > > > > > > > > [3] 
> > > > > > > > > > > > https://github.com/tillrohrmann/flink/tree/autoscaling
> > > > > > > > > > > > [4]
> > > > > > > > > > >
> > > > > > > >
> > > > > https://drive.google.com/file/d/1Vp8W-7Zk_iKXPTAiBT-eLPmCMd_I57Ty/view
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > D.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best
> > > > > > > > > > >
> > > > > > > > > > > ConradJam
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >

Reply via email to