[
https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-13247:
----------------------------------
Priority: Major (was: Minor)
> Implement external shuffle service for YARN
> -------------------------------------------
>
> Key: FLINK-13247
> URL: https://issues.apache.org/jira/browse/FLINK-13247
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: MalcolmSanders
> Priority: Major
>
> Flink batch job users could achieve better cluster utilization and job
> throughput throught external shuffle service because the producers of
> intermedia result partitions can be released once intermedia result
> partitions have been persisted on disks. In
> [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang]
> has introduced pluggable shuffle manager architecture which abstracts the
> process of data transfer between stages from flink runtime as shuffle
> service. I propose to YARN implementation for flink external shuffle service
> since YARN is widely used in various companies.
> The basic idea is as follows:
> (1) Producers write intermedia result partitions to local disks assigned by
> NodeManager;
> (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary
> service, are acknowledged of intermedia result partition descriptions by
> producers;
> (3) Consumers fetch intermedia result partition from yarn shuffle servers;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
