Hi Michaël,
Thanks for mentioning this, users can benefit from this for sure.
This API is not exposed (although there is some code prepared for this)
because community believes that it would require more extensive design and
discussion before the implementation, basically another FLIP.
Although
Hello,
>From the scope section of this FLIP, I understand no API is currently
available to describe resource requirements for DataStream API. Since I need
it for my use case, it seems to be a good opportunity for me to start
contributing to Flink.
Is this task suitable for a new comer ? I am an e
Thanks all for joining the discussion.
It seems to me that there is a consensus on the current FLIP document. So
if there is no objection, I would like to start the voting process for this
FLIP.
Thank you~
Xintong Song
On Wed, Sep 4, 2019 at 8:23 PM Andrey Zagrebin wrote:
> Thanks for updati
Thanks for updating the FLIP Xintong. It looks good to me. I would be ok to
start a vote for it.
Best,
Andrey
On Wed, Sep 4, 2019 at 10:03 AM Xintong Song wrote:
> @all
>
> The FLIP document [1] has been updated.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
>
> https://cwiki.apache.org/confluence/
@all
The FLIP document [1] has been updated.
Thank you~
Xintong Song
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
On Tue, Sep 3, 2019 at 7:20 PM Zhu Zhu wrote:
> Thanks Xintong for the explanation.
>
> For question #1, I think it
Thanks Xintong for the explanation.
For question #1, I think it's good as long as DataSet job behaviors remains
the same.
For question #2, agreed that the resource difference is small enough(at
most 1 edge diff) in current supported point-wise execution edge connection
patterns.
Thanks,
Zhu Zhu
Thanks for the comments, Zhu & Kurt.
Andrey and I also had some discussions offline, and I would like to first
post a summary of our discussion:
1. The motivation of the fraction based approach is to unify resource
management for both operators with specified and unknown resource
requir
Thanks Xingtong for driving this effort, I haven't finished the whole
document yet,
but have couple of questions:
1. Regarding to network memory, the document said it will be derived by
framework
automatically. I'm wondering whether we should delete this dimension from
user-
facing API?
2. Regard
Thanks Xintong for proposing this improvement. Fine grained resources can
be very helpful when user has good planning on resources.
I have a few questions:
1. Currently in a batch job, vertices from different regions can run at the
same time in slots from the same shared group, as long as they do
Updated the FLIP wiki page [1], with the following changes.
- Remove the step of converting pipelined edges between different slot
sharing groups into blocking edges.
- Set `allSourcesInSamePipelinedRegion` to true by default.
Thank you~
Xintong Song
On Mon, Sep 2, 2019 at 11:50 AM X
Regarding changing edge type, I think actually we don't need to do this for
batch jobs neither, because we don't have public interfaces for users to
explicitly set slot sharing groups in DataSet API and SQL/Table API. We
have such interfaces in DataStream API only.
Thank you~
Xintong Song
On T
Thanks for the correction, Till.
Regarding your comments:
- You are right, we should not change the edge type for streaming jobs.
Then I think we can change the option 'allSourcesInSamePipelinedRegion' in
step 2 to 'isStreamingJob', and implement the current step 2 before the
current step 1 so we
Thanks for creating the implementation plan Xintong. Overall, the
implementation plan looks good. I had a couple of comments:
- What will happen if a user has defined a streaming job with two slot
sharing groups? Would the code insert a blocking data exchange between
these two groups? If yes, then
I guess there is a typo since the link to the FLIP-53 is
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
Cheers,
Till
On Tue, Aug 27, 2019 at 1:42 PM Xintong Song wrote:
> Added implementation steps for this FLIP on the wiki page [1].
>
>
>
Added implementation steps for this FLIP on the wiki page [1].
Thank you~
Xintong Song
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
On Mon, Aug 19, 2019 at 10:29 PM Xintong Song wrote:
> Hi everyone,
>
> As Till suggested, t
Hi everyone,
As Till suggested, the original "FLIP-53: Fine Grained Resource Management"
splits into two separate FLIPs,
- FLIP-53: Fine Grained Operator Resource Management [1]
- FLIP-56: Dynamic Slot Allocation [2]
We'll continue using this discussion thread for FLIP-53. For FLIP-56, I
j
Thinks for the comments, Yang.
Regarding your questions:
1. How to calculate the resource specification of TaskManagers? Do they
>have them same resource spec calculated based on the configuration? I
> think
>we still have wasted resources in this situation. Or we could start
>Task
Hi Xintong,
Thanks for your detailed proposal. I think many users are suffering from
waste of resources. The resource spec of all task managers are same and we
have to increase all task managers to make the heavy one more stable. So we
will benefit from the fine grained resource management a lot.
Thanks for the feedbacks, Yangze and Till.
Yangze,
I agree with you that we should make scheduling strategy pluggable and
optimize the strategy to reduce the memory fragmentation problem, and
thanks for the inputs on the potential algorithmic solutions. However, I'm
in favor of keep this FLIP foc
Hi Xintong,
thanks for drafting this FLIP. I think your proposal helps to improve the
execution of batch jobs more efficiently. Moreover, it enables the proper
integration of the Blink planner which is very important as well.
Overall, the FLIP looks good to me. I was wondering whether it wouldn't
Hi, Xintong
Thanks to propose this FLIP. The general design looks good to me, +1
for this feature.
Since slots in the same task executor could have different resource
profile, we will
meet resource fragment problem. Think about this case:
- request A want 1G memory while request B & C want 0.5G
Hi everyone,
We would like to start a discussion thread on "FLIP-53: Fine Grained
Resource Management"[1], where we propose how to improve Flink resource
management and scheduling.
This FLIP mainly discusses the following issues.
- How to support tasks with fine grained resource requirements.
22 matches
Mail list logo
