[
https://issues.apache.org/jira/browse/FLINK-30938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufan Sheng updated FLINK-30938:
--------------------------------
Attachment: (was: image-2023-02-21-00-45-21-712.png)
> Release Testing: Verify FLINK-29766 Adaptive Batch Scheduler should also work
> with hybrid shuffle mode
> ------------------------------------------------------------------------------------------------------
>
> Key: FLINK-30938
> URL: https://issues.apache.org/jira/browse/FLINK-30938
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.17.0
> Reporter: Weijie Guo
> Assignee: Yufan Sheng
> Priority: Blocker
> Labels: release-testing
> Attachments: flink-1.17-branch-log.png, flink-dashborad-version.png,
> taskmanager-slots.png, testAdaptiveBatchJob, testSpeculativeExecution
>
>
> This ticket aims for verifying FLINK-29766: Adaptive Batch Scheduler should
> also work with hybrid shuffle mode.
> More details about this feature and how to use it can be found in this
> [documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle].
> The verification is divided into two parts:
> Part I: Verify hybrid shuffle can work with AdaptiveBatchScheduler
> Write a simple Flink batch job using hybrid shuffle mode and submit this job.
> Note that in flink-1.17, AdaptiveBatchScheduler is the default scheduler for
> batch job, so you do not need other configuration.
> Suppose your job's topology like source -> map -> sink, if your cluster have
> enough slots, you should find that source and map are running at the same
> time.
> Part II: Verify hybrid shuffle can work with Speculative Execution
> Write a Flink batch job using hybrid shuffle mode which has a subtask running
> much slower than others (e.g. sleep indefinitely if it runs on a certain
> host, the hostname can be retrieved via
> InetAddress.getLocalHost().getHostName(), or if its (subtaskIndex +
> attemptNumer) % 2 == 0)
> Modify Flink configuration file to enable speculative execution and tune the
> configuration as you like
> Submit the job. Checking the web UI, logs, metrics and produced result.
> You should find that once a producer task's one subtask finished, all its
> consumer tasks can be scheduled in log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
