[
https://issues.apache.org/jira/browse/FLINK-28926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600256#comment-17600256
]
Yu Chen commented on FLINK-28926:
---------------------------------
Hi, all.
According to the docs of [Batch
Shuffle|https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/],
I have tested the feature in a standalone cluster with 1JM+2TM(2 slots/TM)
locally and submitted the {{WordCount}} example job to the cluster in batch
mode with the following command respectively：
{code:sh}
./bin/flink run -Dexecution.batch-shuffle-mode=ALL_EXCHANGES_BLOCKING
-Dparallelism.default=2 --detached examples/streaming/WordCount.jar --input
/tmp/wordcount.txt --output /tmp/wordcount_res
./bin/flink run -Dexecution.batch-shuffle-mode=ALL_EXCHANGES_HYBRID_FULL
-Dparallelism.default=2 --detached examples/streaming/WordCount.jar --input
/tmp/wordcount.txt --output /tmp/wordcount_res
./bin/flink run -Dexecution.batch-shuffle-mode=ALL_EXCHANGES_HYBRID_SELECTIVE
-Dparallelism.default=2 --detached examples/streaming/WordCount.jar --input
/tmp/wordcount.txt --output /tmp/wordcount_res
{code}
Note that {{/tmp/wordcount.txt}} contains {{10,000,000}} random words generated
by {{{}RandomStringUtils.randomAlphabetic(10){}}}.
Through the Flink WEB, I verified the following scenarios:
# {{HYBRID SHUFFLE}} can utilize all four slots, while {{BLOCK SHUFFLE}} uses
only two slots, which is consistent with the relevant description in the
document.
# By examining the timeline chart of the job, we can see that {{HYBRID
SHUFFLE}} makes the upstream task and the downstream task in the {{WordCount}}
job start simultaneously.
# I also restart the cluster with the configuration `{{{}jobmanager.scheduler:
AdaptiveBatch{}}}` in the {{{}flink-conf.yaml{}}}. The job configured with
{{ALL_EXCHANGES_HYBRID_FULL}} and {{ALL_EXCHANGES_HYBRID_SELECTIVE}} produced
an error consistent with the documentation's description of limitation.
Overall, I have not found any problems with this feature, please feel free to
contact me if any other cases need to be tested.
> Release Testing: Verify flip-235 hybrid shuffle mode
> ----------------------------------------------------
>
> Key: FLINK-28926
> URL: https://issues.apache.org/jira/browse/FLINK-28926
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.16.0
> Reporter: Weijie Guo
> Assignee: Yu Chen
> Priority: Blocker
> Labels: release-testing
> Fix For: 1.16.0
>
>
> * Please refer to release note of FLINK-27862 for a list of changes need to
> be verified.
> * Please refer to out document for more details
> [https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle|https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/]
> * Hybrid shuffle have some known limitations: No support for Slot Sharing,
> Adaptive Batch Scheduler and Speculative Execution. Please make sure you do
> not using this features in testing.
> * The changes should be verified only in batch execution mode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
