nzw921rx opened a new pull request, #13196:
URL: https://github.com/apache/seatunnel/pull/13196
### Purpose of this pull request
This contribution to Apache SeaTunnel reduces queue handoff overhead in
`MultiTableSinkWriter`. With HASH routing and fast consumers, adding replicas
can increase how often workers block on empty queues and producers wake them on
the next enqueue. The throughput drop is reproducible with zero simulated sink
delay in the benchmark added by #13162.
When a multi-replica worker observes its queue drain again within 200 µs, it
requests a 50 µs pause before polling again. This gives incoming rows a brief
window to collect in the existing bounded FIFO and reduces repeated wakeups.
The worker pauses after processing the row and updating the pending-row
count. Data rows and schema-change barriers share the same FIFO. Full queues
continue to apply backpressure, and checkpoint draining continues to track
accepted rows.
### Does this PR introduce _any_ user-facing change?
This is an internal scheduling change that uses the existing configuration
and queue capacity.
### How was this patch tested?
Ran `MultiTableSinkBenchmark.recordHandoff` in baseline → patch → patch →
baseline order against `b7ef493`. Both revisions used Temurin 11.0.33 on the
same four pinned logical CPUs, with one producer, one table, HASH routing, and
zero simulated sink delay. The table shows the mean of the two rounds per
revision.
| Replicas | Baseline (ops/s) | Patch (ops/s) | Speedup |
|---:|---:|---:|---:|
| 1 | 4,049,218 | 5,431,414 | Control |
| 5 | 379,491 | 7,569,318 | 19.95× |
| 15 | 363,121 | 7,235,015 | 19.92× |
Coalescing applies only with multiple replicas. The benchmark measures
in-memory handoff and checks per-replica delivery counts and ordering at
teardown.
```bash
./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests package
taskset -c 0-3 java -jar seatunnel-benchmarks/target/benchmarks.jar \
MultiTableSinkBenchmark.recordHandoff \
-p replicaNum=1,5,15 -p tableCount=1 -p routing=HASH \
-p consumerBatchSize=1000 -p batchDelayMicros=0 -p recordPoolSize=65536 \
-t 1 -f 1 -wi 2 -w 1s -i 4 -r 1s -foe true \
-jvmArgsAppend '-Xms1g -Xmx1g -XX:ActiveProcessorCount=4'
```
Formatting and benchmark reactor verification passed. All 45 targeted tests
passed across `MultiTableSinkWriterTest`,
`MultiTableSinkWriterSchemaChangeBroadcastTest`, and
`MultiTableSinkFixtureTest`, including checkpoint draining, schema barriers,
and trailing rows.
At 5,000 rows/s across 15 replicas, one short-warmup run reached 3.4 ms P99
with the patch. With a longer warmup, P99 was 0.25–0.26 ms with the patch and
0.23–0.36 ms on the baseline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]