avantgardnerio opened a new pull request, #26861: URL: https://github.com/apache/datafusion/pull/26861
## Which issue does this PR close? - Part of #25650. ## Rationale for this change #25651 logs drift between `MemoryPool` reservations and allocated bytes in SLT, but test files run concurrently, so it compares the whole process against every file's reservations and the file it names can be wrong. This charges each file's allocations to its own account, so drift is measured per file without `--test-threads 1`. DataFusion still does not choose the allocator: the per-thread count comes from a caller-supplied `fn() -> u64` (here SLT's existing `CountingAllocator`; jemalloc's `thread.allocatedp`/`deallocatedp` work too). ## What changes are included in this PR? - `datafusion-common-runtime`: `MemoryAccount`, `MemoryTrackedFuture` (charges each poll the change in the per-thread count), and `MemoryAccountTracer`, a `JoinSetTracer` that charges tasks spawned through `SpawnedTask`/`JoinSet` to the spawner's account. - `sqllogictest`: each file runs in its own account with its own `MemoryDriftTracker`. `CountingAllocator` keeps an exact per-thread count instead of flushing to a global one. - `sqllogictest`: `AccountedObjectStore` wraps `file://`, because `LocalFileSystem` allocates and frees on its own `spawn_blocking` threads, which no account charges. It keeps written payloads alive until the write finishes and charges ranges it reads. Without it, a 10M row `COPY` left the 25.6 MB it wrote on the file's account. Peak drift per file, full suite, 4 MB log threshold: | File | Peak drift | Allocated | Reserved | Sampled at | |---|---|---|---|---| | `push_down_filter_regression.slt` | 43.3 MB | 43.6 MB | 344.5 KB | `ParquetSink(SerializedFileWriter)` | | `piecewise_merge_join_nested_nulls.slt` | 29.8 MB | 29.9 MB | 147.0 KB | `ExternalSorterMerge[0]` | | `cte_materialized.slt` | 15.3 MB | 15.3 MB | 24 B | `MaterializedCte[c][0]` | | `aggregate.slt` | 12.5 MB | 15.9 MB | 3.4 MB | `RepartitionExec[0]` | | `parquet.slt` | 11.6 MB | 11.6 MB | 72.8 KB | `ParquetSink(SerializedRowGroupWriter)` | | `cte.slt` | 11.2 MB | 11.2 MB | 24.3 KB | `ExternalSorterMerge[3]` | | `push_down_filter_parquet.slt` | 9.1 MB | 9.1 MB | 32.0 KB | `ExternalSorterMerge[0]` | | `imdb.slt` | 7.8 MB | 8.2 MB | 329.3 KB | `HashJoinInput` | | `joins.slt` | 7.4 MB | 7.4 MB | 160 B | `AggregateStream[0]` | | `clickbench.slt` | 6.8 MB | 6.8 MB | 0 B | `RepartitionExec[0]` | | `window.slt` | 5.8 MB | 7.4 MB | 1673.8 KB | `SortPreservingMergeExec[0]` | | `group_by.slt` | 5.8 MB | 6.5 MB | 707.8 KB | `ExternalSorterMerge[2]` | The consumer is the reservation change that took the sample, not necessarily what allocated the memory. ## What is the testing strategy for this PR? Unit tests in `memory_account.rs` cover charging, nesting, reading mid-poll, and the tracer. Accounts balance at file end: a plain scan nets 32 KB, a 10M row `COPY` 217 KB, and `push_down_filter_regression.slt` 1.8 MB. Known gap: reads through `GetResult::bytes()`/`into_stream()` on a local file still allocate on object_store's threads, so drift can under-report there. ## Are there any user-facing changes? New public API in `datafusion-common-runtime`. Drift logging still only logs and never fails a test. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
