Hi All,
Thanks for all the feedback about this FLIP.
Since there are no other concerns, this FLIP-235 discussion is over. I will
open a vote today.
Best regards,
Weijie
Xintong Song 于2022年5月25日周三 22:17写道：
> Ok, I think we are on the same page. I'm aware of
> ExecutionConfig#setExecutionMod
Ok, I think we are on the same page. I'm aware of
ExecutionConfig#setExecutionMode, which sets the data exchanging mode at
the job scope.
Best,
Xintong
On Wed, May 25, 2022 at 9:50 PM Chesnay Schepler wrote:
> You can influence it to some extent via ExecutionConfig#setExecutionMode.
> You ca
You can influence it to some extent via ExecutionConfig#setExecutionMode.
You can for example for all shuffles to use blocking exchanges.
I'm not proposing an API that would allow this to be set per edge.
On 25/05/2022 15:23, Xintong Song wrote:
In general, I agree with you about aiming jobs wit
In general, I agree with you about aiming jobs with no/few blocking
exchanges for fine-grained recovery. The only problem is, correct me if I'm
wrong, users currently cannot control the data exchanging mode of a
specific edge. I'm not aware of such APIs.
As a first step, I'd prefer excluding this
Yes; but that's also a limitation of the current fine-grained recovery.
My suggestion was primarily aimed at jobs that have no/few blocking
exchanges, where users would currently have to explicitly configure
additional blocking exchanges to really get something out of
fine-grained recovery (at
>
> Will this also allow spilling everything to disk while also forwarding
> data to the next task?
>
Yes, as long as the downstream task is started, this always forward the
data, even while spilling everything.
This would allow us to improve fine-grained recovery by no longer being
> constrained
Will this also allow spilling everything to disk while also forwarding
data to the next task?
This would allow us to improve fine-grained recovery by no longer being
constrained to pipelined regions.
On 25/05/2022 05:55, weijie guo wrote:
Hi All,
Thank you for your attention and feedback.
Do
Hi All,
Thank you for your attention and feedback.
Do you have any other comments? If there are no other questions, I'll vote
on FLIP-235 tomorrow.
Best regards,
Weijie
Aitozi 于2022年5月20日周五 13:22写道：
> Hi Xintong
> Thanks for your detailed explanation, I misunderstand the spill
> behavior
Hi Xintong
Thanks for your detailed explanation, I misunderstand the spill
behavior at first glance,
I get your point now. I think it will be a good addition to the current
execution mode.
Looking forward to it :)
Best,
Aitozi
Xintong Song 于2022年5月20日周五 12:26写道：
> Hi Aitozi,
>
> In which ca
Hi Aitozi,
In which case we can use the hybrid shuffle mode
Just to directly answer this question, in addition to
Weijie's explanations. For batch workload, if you want the workload to take
advantage of as many resources as available, which ranges from a single
slot to as many slots as the total
Thanks Weijie for your answer. So you mean the hybrid shuffle mode will
limit
its usage to the bounded source, Right ?
One more question, with the bounded data and partly of the stage is running
in the Pipelined shuffle mode, what will be the behavior of the task
failure, Is the
checkpoint enabled
Hi, Aitozi:
Thank you for the feedback!
Here are some of my thoughts on your question
>>> 1.If there is an unbounded data source, but only have resource to
schedule the first stage, will it bring the big burden to the disk/shuffle
service which will occupy all the resource I think.
First of all,
Hi Weijie:
Thanks for the nice FLIP, I have couple questions about this:
1) In the hybrid shuffle mode, the shuffle mode is decided by the resource.
If there
is an unbounded data source, but only have resource to schedule the first
stage, will it
bring the big burden to the disk/shuffle serv
Yangze, Thank you for the feedback!
Here's my thoughts for your questions:
>>> How do we decide the size of the buffer pool in MemoryDataManager and
the read buffers in FileDataManager?
The BufferPool in MemoryDataManager is the LocalBufferPool used by
ResultPartition, and the size is the same as
Thanks for driving this. Xintong and Weijie.
I believe this feature will make Flink a better batch/OLAP engine. +1
for the overall design.
Some questions:
1. How do we decide the size of the buffer pool in MemoryDataManager
and the read buffers in FileDataManager?
2. Is there an upper limit for t
Thanks for preparing this FLIP, Weijie.
I think this is a good improvement on batch resource elasticity. Looking
forward to the community feedback.
Best,
Xintong
On Thu, May 19, 2022 at 2:31 PM weijie guo
wrote:
> Hi all,
>
>
> I’d like to start a discussion about FLIP-235[1], which introdu
16 matches
Mail list logo
