> > > > > returned
> > > > > >> > So maybe in the first step, let's mark batch jobs with PIPELINED
> > > > > edges as an invalid case for fine-grained resources and throw
> > > exception for
> > > > > it in early comp
t; it in early compiling stage?
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Zhu
> > > > >> >
> > > > >> > Yangze Guo 于2021年6月15日周二 下午4:57写道：
> > > > >> >>
> > >
27;ve annotated these
> two
> > > >> >> points in the FLIP.
> > > >> >> The vote is now started in [1].
> > > >> >>
> > > >> >> [1]
> > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-169-DataStream-
t; >> Yangze Guo
> > >> >>
> > >> >> On Fri, Jun 11, 2021 at 2:50 PM Yun Gao
> > wrote:
> > >> >> >
> > >> >> > Hi,
> > >> >> >
> > >> >> > Very thanks @Yangze for bringin
gt;> > Hi,
> >> >> >
> >> >> > Very thanks @Yangze for bringing up this discuss. Overall +1 for
> >> >> > exposing the fine-grained resource requirements in the DataStream
> API.
> >> >> >
> >> >> > One si
t; >
>> >> > Hi,
>> >> >
>> >> > Very thanks @Yangze for bringing up this discuss. Overall +1 for
>> >> > exposing the fine-grained resource requirements in the DataStream API.
>> >> >
>> >> > One similar issue
> resources. We might need to do some check internally. But We could
> also
> >> > leave that during the development of the actual PR.
> >> >
> >> > Best,
> >> > Yun
> >> >
> >> >
> >> >
But We could also
>> > leave that during the development of the actual PR.
>> >
>> > Best,
>> > Yun
>> >
>> >
>> >
>> > --Original Mail --
>> > Sender:Arvid Heise
>> > Send Date:Thu
me check internally. But We could also
> > leave that during the development of the actual PR.
> >
> > Best,
> > Yun
> >
> >
> >
> > ----------Original Mail ------
> > Sender:Arvid Heise
> > Send Date:Thu Jun 10 15:33:37 2021
> > Recip
-
> Sender:Arvid Heise
> Send Date:Thu Jun 10 15:33:37 2021
> Recipients:dev
> Subject:Re: [DISCUSS] FLIP-169: DataStream API for Fine-Grained Resource
> Requirements
> Hi Yangze,
>
>
>
> Thanks for incorporating the ideas and sorry for missing the builder part.
: DataStream API for Fine-Grained Resource
Requirements
Hi Yangze,
Thanks for incorporating the ideas and sorry for missing the builder part.
My main idea is that SlotSharingGroup is immutable, such that the user
doesn't do:
ssg = new SlotSharingGroup();
ssg.setCpus(2);
operator1.slotSharing
Hi Yangze,
Thanks for incorporating the ideas and sorry for missing the builder part.
My main idea is that SlotSharingGroup is immutable, such that the user
doesn't do:
ssg = new SlotSharingGroup();
ssg.setCpus(2);
operator1.slotSharingGroup(ssg);
ssg.setCpus(4);
operator2.slotSharingGroup(ssg);
Thanks all for the discussion. I've updated the FLIP accordingly, the
key changes are:
- Introduce SlotSharingGroup instead of ResourceSpec which contains
the resource spec of slot sharing group
- Introduce two interfaces for specifying the SlotSharingGroup:
#slotSharingGroup(SlotSharingGroup) and
Thanks for the valuable suggestion, Arvid.
1) Yes, we can add a new SlotSharingGroup which includes the name and
its resource. After that, we have two interfaces for configuring the
slot sharing group of an operator:
- #slotSharingGroup(String name)// the resource of it can be
configured throu
Hi Yangze,
I like the general approach to bind requirements to slotsharing groups. I
think the current approach is also flexible enough that a user could simply
use ParameterTool or similar to use config values and wire that with their
slotgroups, such that different requirements can be tested wit
@Yang
In short, the external resources will participate in resource
deduction and be logically ensured, but requesting an external
resource must still be done through config options with the current
default resource allocation strategy.
In FLIP-56, we abstract the logic of resource allocation to th
Thanks @Yangze for preparing this FLIP.
I think this is a good start point for the community users to have a taste
on the fine-grained
resource management, which we all believe it could improve the Flink job
stability and
cluster utilization.
I have a simple question about the extended resources.
@Xintong
> introduce a general approach for overwriting such job specifics without
> re-compiling the job
I think that would be a good direction. Just share some cents on this
topic. I'd divide the job-level specifics into two categories:
- Specifics which affect how Flink executes the job, e.g.
"
I think being able to specify fine grained resource requirements without
having to change the codes and recompile the job is indeed a good idea. It
definitely improves the usability.
However, this requires more careful designs, which probably deserves a
separate thread. I'd be good to have that di
@Wenlong
After another consideration, the config option approach I mentioned
above might not be appropriate. The resource requirements for SSG
should be a job level configuration and should no be set in the
flink-conf.
I think we can define a JSON format, which would be the ResourceSpecs
mapped by
Thanks for the feedbacks, Xintong and Wenlong!
@Wenlong
I think that is a good idea, adjust the resource without re-compiling
the job will facilitate the tuning process.
We can define a pattern "slot-sharing-group.resource.{ssg name}"
(welcome any proposal for the prefix naming) for the resource s
Thanks Yangze for the flip, it is great for users to be able to declare the
fine-grained resource requirements for the job.
I have one minor suggestion: can we support setting resource requirements
by configuration? Currently most of the config options in execution config
can be configured by conf
Thanks Yangze for preparing the FLIP.
The proposed changes look good to me.
As you've mentioned in the implementation plan, I believe one of the most
important tasks of this FLIP is to have the feature well documented. It
would be really nice if we can keep that in mind and start drafting the
doc
Hi, there,
We would like to start a discussion thread on "FLIP-169: DataStream
API for Fine-Grained Resource Requirements"[1], where we propose the
DataStream API for specifying fine-grained resource requirements in
StreamExecutionEnvironment.
Please find more details in the FLIP wiki document [1
24 matches
Mail list logo
