[
https://issues.apache.org/jira/browse/FLINK-33742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuxin Tan updated FLINK-33742:
------------------------------
Description:
At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes
features such as Dynamic Partition Pruning (DPP), Runtime Filter, and Adaptive
Batch Scheduler, are not fully compatible. While they can be used concurrently
at the same time, the activation of AQE inhibits the key capability of Hybrid
Shuffle to perform simultaneous reading and writing. This limitation arises
because AQE dictates that downstream tasks may only initiate once upstream
tasks have finished, a requirement that is inconsistent with the simultaneous
read-write process facilitated by Hybrid Shuffle. In addition, Hybrid Shuffle
will restart the whole job when failover, which is also an essential issue for
production usage.
To harness the full potential of Hybrid Shuffle and AQE, it is essential to
refine their integration. By doing so, we can capitalize on each feature's
distinct advantages and enhance overall system performance.
was:
At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes
features such as Dynamic Partition Pruning (DPP), Runtime Filter, and Adaptive
Batch Scheduler, are not fully compatible. While they can be used concurrently
at the same time, the activation of AQE inhibits the key capability of Hybrid
Shuffle to perform simultaneous reading and writing. This limitation arises
because AQE dictates that downstream tasks may only initiate once upstream
tasks have finished, a requirement that is inconsistent with the simultaneous
read-write process facilitated by Hybrid Shuffle.
To harness the full potential of Hybrid Shuffle and AQE, it is essential to
refine their integration. By doing so, we can capitalize on each feature's
distinct advantages and enhance overall system performance.
> Hybrid Shuffle should work well with Adaptive Query Execution
> -------------------------------------------------------------
>
> Key: FLINK-33742
> URL: https://issues.apache.org/jira/browse/FLINK-33742
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
> Affects Versions: 1.19.0
> Reporter: Yuxin Tan
> Priority: Major
> Labels: Umbrella
>
> At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes
> features such as Dynamic Partition Pruning (DPP), Runtime Filter, and
> Adaptive Batch Scheduler, are not fully compatible. While they can be used
> concurrently at the same time, the activation of AQE inhibits the key
> capability of Hybrid Shuffle to perform simultaneous reading and writing.
> This limitation arises because AQE dictates that downstream tasks may only
> initiate once upstream tasks have finished, a requirement that is
> inconsistent with the simultaneous read-write process facilitated by Hybrid
> Shuffle. In addition, Hybrid Shuffle will restart the whole job when
> failover, which is also an essential issue for production usage.
> To harness the full potential of Hybrid Shuffle and AQE, it is essential to
> refine their integration. By doing so, we can capitalize on each feature's
> distinct advantages and enhance overall system performance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
