FYI, I updated the FLIP accordingly. To sum up, Flink will throw an
exception and tell user to configure an internal
"fine-grained.shuffle-mode.all-blocking" to be true in this scenario.
Best,
Yangze Guo
On Tue, Jun 22, 2021 at 2:20 PM Yangze Guo wrote:
>
> Thanks for the comment, Xintong.
>
> I
Thanks for the comment, Xintong.
I used to wonder if it was reasonable or worthwhile to introduce a
configuration like "table.exec.shuffle-mode" for DataStream API.
Narrow down the scope of effect sounds good to me.
Best,
Yangze Guo
On Tue, Jun 22, 2021 at 2:08 PM Xintong Song wrote:
>
> I seco
I second Zhu and Till's opinion.
Failing with an exception that also includes how to resolve the problem
sounds better, in terms of making it explicit to users that pipelined edges
are replaced with blocking edges.
Concerning absence of knobs tuning the edge types, we can introduce a
configuratio
Thanks for the feedback, Till!
Actually, we cannot give user any resolution for this issue as there
is no API for DataStream users to influence the edge types at the
moment. The edge types are currently fixed based on the jobs' mode
(batch or streaming).
a) I think it might not confuse the user a
I would be more in favor of what Zhu Zhu proposed to throw an exception
with a meaningful and understandable explanation that also includes how to
resolve this problem. I do understand the reasoning behind automatically
switching the edge types in order to make things easier to use but a) this
can
Thanks, I append it to the known limitations of this FLIP.
Best,
Yangze Guo
On Mon, Jun 21, 2021 at 3:20 PM Zhu Zhu wrote:
>
> Thanks for the quick response Yangze.
> The proposal sounds good to me.
>
> Thanks,
> Zhu
>
> Yangze Guo 于2021年6月21日周一 下午3:01写道：
>>
>> Thanks for the comments, Zhu!
>>
Thanks for the quick response Yangze.
The proposal sounds good to me.
Thanks,
Zhu
Yangze Guo 于2021年6月21日周一 下午3:01写道：
> Thanks for the comments, Zhu!
>
> Yes, it is a known limitation for fine-grained resource management. We
> also have filed this issue in FLINK-20865 when we proposed FLIP-156.
Thanks for the comments, Zhu!
Yes, it is a known limitation for fine-grained resource management. We
also have filed this issue in FLINK-20865 when we proposed FLIP-156.
As a first step, I agree that we can mark batch jobs with PIPELINED
edges as an invalid case for this feature. However, just th
Thanks for proposing this @Yangze Guo and sorry for
joining the discussion so late.
The proposal generally looks good to me. But I find one problem that batch
job with PIPELINED edges might hang if enabling fine-grained resources. see
"Resource Deadlocks could still happen in certain Cases" sectio
Thanks for the supplement, Arvid and Yun. I've annotated these two
points in the FLIP.
The vote is now started in [1].
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-169-DataStream-API-for-Fine-Grained-Resource-Requirements-td51381.html
Best,
Yangze Guo
On Fri, Jun
Hi,
Very thanks @Yangze for bringing up this discuss. Overall +1 for
exposing the fine-grained resource requirements in the DataStream API.
One similar issue as Arvid has pointed out is that users may also creating
different SlotSharingGroup objects, with different names but with different
reso
11 matches
Mail list logo
