Sruhvx-jpg opened a new pull request, #26843:
URL: https://github.com/apache/datafusion/pull/26843
# Which issue does this PR close?
Closes part of #26759.
# Rationale for this change
In CI merge queues, `leaf_pushdown_fuzz_extended` bottlenecks 32-vCPU
runners for 229–238 seconds by sequentially evaluating 5,000 in-memory and
1,000 Parquet differential queries on a single thread.
# What changes are included in this PR?
1. **Dynamic Work-Stealing Pool (`Arc<AtomicU64>`)**:
- Spawns worker tasks scaled to host vCPUs via `available_parallelism()`,
clamped to 16 workers to avoid thrashing against DataFusion's internal
`target_partitions = 2` plans.
- Workers dynamically claim seeds via `fetch_add(1, Ordering::Relaxed)`
with a pre-check `load(Relaxed)` guard, eliminating chunk-slicing math and
straggler delays.
2. **Safe Concurrent Contexts**:
- `SessionContext` is safely cloned across Tokio tasks since schemas and
tables are pre-registered and queries only take read locks.
3. **Deterministic Aggregation & Typed Newtypes**:
- Workers record local metrics and their earliest failing seed. The
coordinator reduces by minimal seed number, preserving 100% deterministic
failure reporting and reproduction (`LEAF_PUSHDOWN_FUZZ_SEED=<seed>
LEAF_PUSHDOWN_FUZZ_CASES=1`).
- Introduces `TestEngine`, `Cases`, and `Seed` newtypes to prevent
parameter transposition.
# Are these changes tested?
Yes, tested against `extended_tests`.
### Benchmark Results
- **`leaf_pushdown_fuzz_extended` (6,000 cases)**:
- Baseline (sequential): **141.16 s** (local) / **~238 s** (CI)
- Redesign (work-stealing): **15.20 s** (**~9.3x - 15.6x speedup**)
- `hyperfine` (3 runs): **15.917 s ± 0.531 s**
- **Hardware Footprint**:
- CPU Saturation: **1522%** (16 worker threads fully utilized)
- Peak RSS: **153.2 MiB** (+19.5 MiB over single-thread baseline)
- **Correctness**: 6,000 / 6,000 cases passed (0 skipped, 0 failed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
