Sruhvx-jpg commented on issue #26759:
URL: https://github.com/apache/datafusion/issues/26759#issuecomment-6410897727
I'd like to pick up `leaf_pushdown_fuzz_extended` in a dedicated PR to
address the ~235s extended CI bottleneck. (Researched the test harness with AI
assistance).
**Proposed Plan:**
1. **Dynamic Seed Partitioning:**
- Add a helper function to divide `cases` into disjoint `(seed_start,
count)` chunks based on `std::thread::available_parallelism()`.
- Scale workers dynamically from local dev up to CI, clamped to `1..=16`
(to prevent memory/allocator thrashing from nested `target_partitions = 2`
plans), bounded by `min(cases)` so single-seed debugging
(`LEAF_PUSHDOWN_FUZZ_CASES=1`) spawns exactly one task.
- Distribute division remainders evenly across chunks.
2. **Concurrent Execution:**
- Run each chunk in parallel via `tokio::spawn`.
- In-memory tables and Parquet `TempDir` files are pre-registered and
strictly read-only, so `SessionContext` can be safely cloned across worker
tasks without state conflicts.
3. **Deterministic Aggregation:**
- Each worker reports its `skipped` count, `failed` count, and earliest
failing seed.
- The coordinator sums the counts and preserves the lowest failing seed
across all workers, keeping failure reporting and reproduction
(`LEAF_PUSHDOWN_FUZZ_SEED=<seed>`) 100% deterministic.
Does this design look sound to you, or are there any caveats to consider
before I proceed with the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
