Han Yin created FLINK-41061:
-------------------------------
Summary: Daily benchmark keeps failing because of excessive
runtime of the adaptive partition benchmark
Key: FLINK-41061
URL: https://issues.apache.org/jira/browse/FLINK-41061
Project: Flink
Issue Type: Bug
Components: Benchmarks
Reporter: Han Yin
Attachments: flink-main-benchmarks #9173 Console [Jenkins].html
{{AdaptivePartitionThroughputBenchmark.mapRebalanceMapRescaleMapSink}}
introduces processing skew by sleeping for *1 ms per record* in some subtasks
of the downstream Map operators.
This makes individual benchmark invocations take tens to hundreds of seconds.
With *3 forks × (10 warmup + 10 measurement iterations)* for each of the four
parameter combinations, the accumulated runtime is excessive.
In Jenkins build {*}#9173{*}(as attached below), this benchmark ran for *5h
45m* and was still executing when the pipeline reached its {*}6-hour timeout{*}.
Local testing of the introduction-era version also reproduced long invocations,
so this is {*}not a confirmed recent Flink regression{*}.
h3. Proposed change
Sleep for {*}1 ms every 100 records{*}, rather than every record, using an
independent counter in each slow operator instance.
Keep the existing data volume, parallelism, topology and checkpoint
configuration unchanged. Therefore we can retain artificial processing skew but
accelerate the whole process.
h3. Local validation
Tested with Flink {{{}855597a{}}}, benchmark {{{}123532a{}}}, macOS ARM64 and
JDK 11.0.26.
With {*}1 fork, no warmup and 1 measurement iteration{*}, average invocation
times were
||Adaptive partitioner||Max traverse size||Sleep every record||Sleep every 100
records||
|Disabled|2|220.0 s|2.38 s|
|Enabled|2|58.2 s|1.10 s|
|Enabled|6|39.6 s|0.74 s|
Additional testing with *1 warmup and 3 measurement iterations* showed
* Disabled: *1.94–2.03 s*
* Enabled, traverse size 2: *0.56–0.64 s*
* Enabled, traverse size 6: *0.38–0.41 s*
All runs completed successfully. The reduced sleep frequency substantially
shortened execution while retaining an observable benefit from adaptive
partition selection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
