txwyy123 commented on code in PR #26149:
URL: https://github.com/apache/datafusion/pull/26149#discussion_r4250594944
##########
datafusion/common/src/utils/memory.rs:
##########
@@ -464,46 +586,111 @@ fn array_children(array: &ArrayRef) -> Vec<&ArrayRef> {
}
}
-/// Tracks a small number of buffers inline, avoiding a heap allocation for
-/// typical batches, and promotes to a hash set when more buffers are seen.
+/// Reference-counted buffer tracker with an inline fast path.
+///
+/// Tracks buffer addresses with their reference counts. The first
+/// [`INLINE_BUFFER_IDS`] distinct buffers are stored in a fixed-size inline
+/// array to avoid heap allocation for typical batches. When more buffers are
+/// encountered, the tracker promotes to a heap-allocated `HashMap`.
#[derive(Debug)]
-struct BufferIdSet {
- inline: [Option<NonZero<usize>>; INLINE_BUFFER_IDS],
+struct BufferIdMap {
+ /// Inline storage: `(address, count)` pairs.
+ inline: [(NonZero<usize>, u32); INLINE_BUFFER_IDS],
Review Comment:
Fixed. Split the inline storage from interleaved tuples (256 bytes) into
separate ids and counts arrays (128 + 64 = 192 bytes). The hot position() scan
now touches a contiguous 128-byte address array without interleaved count
words. Will add main vs PR comparison to the description.
##########
datafusion/common/src/utils/memory.rs:
##########
@@ -1103,4 +1290,924 @@ mod record_batch_tests {
let size = get_record_batch_memory_size(&batch);
assert_eq!(size, 8208);
}
+
+ // ---- uncount tests ----
+
+ #[test]
+ fn test_uncount_batch_round_trip() {
Review Comment:
Done. Added the uncount round-trip to assert_array_memory_size_matches
(covers every array layout automatically), collapsed the per-type shared-buffer
tests into one table-driven test_uncount_shared_buffer_pairs (9 cases), and
removed the 17 standalone round-trip tests. Net: -716 / +296 lines (tests went
from ~900 to ~280 lines). Kept: randomized model, overflow promotion,
never-counted, double-uncount, and both overhead tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
