[
https://issues.apache.org/jira/browse/FLINK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-27862:
-------------------------------
Release Note:
We have introduced a new Hybrid Shuffle Mode for batch executions. It combines
the advantages of blocking shuffle and pipelined shuffle (in streaming mode).
- Like blocking shuffle, it does not require upstream and downstream tasks to
run simultaneously, which allows executing a job with little resources.
- Like pipelined shuffle, it does not require downstream tasks to be executed
after upstream tasks finish, which reduces the overall execution time of the
job when given sufficient resources.
- It adapts to custom preferences between persisting less data and restarting
less tasks on failures, by providing different spilling strategies.
For more details:
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/batch/batch_shuffle
was:
We have introduced a new Hybrid Shuffle Mode for batch executions. It combines
the advantages of blocking shuffle and pipelined shuffle (in streaming mode).
- Like blocking shuffle, it does not require upstream and downstream tasks to
run simultaneously, which allows executing a job with little resources.
- Like pipelined shuffle, it does not require downstream tasks to be executed
after upstream tasks finish, which reduces the overall execution time of the
job when given sufficient resources.
- It adapts to custom preferences between persisting less data and restarting
less tasks on failures, by providing different spilling strategies.
For more details:
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/batch/batch_shuffle.md
> FLIP-235: Hybrid Shuffle Mode
> -----------------------------
>
> Key: FLINK-27862
> URL: https://issues.apache.org/jira/browse/FLINK-27862
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: Weijie Guo
> Assignee: Weijie Guo
> Priority: Major
> Labels: Umbrella
> Fix For: 1.16.0
>
>
> Introduce a new shuffle mode can overcome some of the problems of Pipelined
> Shuffle and Blocking Shuffle in batch scenarios, it can make best use of
> available resources and minimize disk IO load.
> More details see
> [FLIP-235|https://cwiki.apache.org/confluence/display/FLINK/FLIP-235%3A+Hybrid+Shuffle+Mode]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
