[
https://issues.apache.org/jira/browse/FLINK-30938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-30938:
-------------------------------
Description:
The document has not been completed, this testing work should start after
FLINK-30860 is completed.
This ticket aims for verifying FLINK-29766: Adaptive Batch Scheduler should
also work with hybrid shuffle mode.
More details about this feature and how to use it can be found in this
[documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle].
The verification is divided into two parts:
Part I: Verify hybrid shuffle can work with AdaptiveBatchScheduler
Write a simple Flink batch job using hybrid shuffle mode and submit this job.
Note that in flink-1.17, AdaptiveBatchScheduler is the default scheduler for
batch job, so you do not need other configuration.
Suppose your job's topology like source -> map -> sink, if your cluster have
enough slots, you should find that source and map are running at the same time.
Part II: Verify hybrid shuffle can work with Speculative Execution
Write a Flink batch job using hybrid shuffle mode which has a subtask running
much slower than others (e.g. sleep indefinitely if it runs on a certain host,
the hostname can be retrieved via InetAddress.getLocalHost().getHostName(), or
if its (subtaskIndex + attemptNumer) % 2 == 0)
Modify Flink configuration file to enable speculative execution and tune the
configuration as you like
Submit the job. Checking the web UI, logs, metrics and produced result.
You should find that once a producer task's one subtask finished, all its
consumer tasks can be scheduled in log.
was:
The document has not been completed, this testing work should start after
FLINK-30860 is completed.
This ticket aims for verifying FLINK-29766: Adaptive Batch Scheduler should
also work with hybrid shuffle mode.
More details about this feature and how to use it can be found in this
[documentation|xxx].
The verification is divided into two parts:
Part I: Verify hybrid shuffle can work with AdaptiveBatchScheduler
Write a simple Flink batch job using hybrid shuffle mode and submit this job.
Note that in flink-1.17, AdaptiveBatchScheduler is the default scheduler for
batch job, so you do not need other configuration.
Suppose your job's topology like source -> map -> sink, if your cluster have
enough slots, you should find that source and map are running at the same time.
Part II: Verify hybrid shuffle can work with Speculative Execution
Write a Flink batch job using hybrid shuffle mode which has a subtask running
much slower than others (e.g. sleep indefinitely if it runs on a certain host,
the hostname can be retrieved via InetAddress.getLocalHost().getHostName(), or
if its (subtaskIndex + attemptNumer) % 2 == 0)
Modify Flink configuration file to enable speculative execution and tune the
configuration as you like
Submit the job. Checking the web UI, logs, metrics and produced result.
You should find that once a producer task's one subtask finished, all its
consumer tasks can be scheduled in log.
> Release Testing: Verify FLINK-29766 Adaptive Batch Scheduler should also work
> with hybrid shuffle mode
> ------------------------------------------------------------------------------------------------------
>
> Key: FLINK-30938
> URL: https://issues.apache.org/jira/browse/FLINK-30938
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.17.0
> Reporter: Weijie Guo
> Assignee: xy
> Priority: Blocker
> Labels: release-testing
> Attachments: testAdaptiveBatchJob, testSpeculativeExecution
>
>
> The document has not been completed, this testing work should start after
> FLINK-30860 is completed.
> This ticket aims for verifying FLINK-29766: Adaptive Batch Scheduler should
> also work with hybrid shuffle mode.
> More details about this feature and how to use it can be found in this
> [documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle].
> The verification is divided into two parts:
> Part I: Verify hybrid shuffle can work with AdaptiveBatchScheduler
> Write a simple Flink batch job using hybrid shuffle mode and submit this job.
> Note that in flink-1.17, AdaptiveBatchScheduler is the default scheduler for
> batch job, so you do not need other configuration.
> Suppose your job's topology like source -> map -> sink, if your cluster have
> enough slots, you should find that source and map are running at the same
> time.
> Part II: Verify hybrid shuffle can work with Speculative Execution
> Write a Flink batch job using hybrid shuffle mode which has a subtask running
> much slower than others (e.g. sleep indefinitely if it runs on a certain
> host, the hostname can be retrieved via
> InetAddress.getLocalHost().getHostName(), or if its (subtaskIndex +
> attemptNumer) % 2 == 0)
> Modify Flink configuration file to enable speculative execution and tune the
> configuration as you like
> Submit the job. Checking the web UI, logs, metrics and produced result.
> You should find that once a producer task's one subtask finished, all its
> consumer tasks can be scheduled in log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
