rich7420 opened a new pull request, #6498: URL: https://github.com/apache/datafusion-comet/pull/6498
## Which issue does this PR close? Closes #6225. ## Rationale for this change When ListExtract has an absent or NULL default, Arrow take can reserve nested child capacity using unselected input lengths. Nested MutableArrayData builders can also propagate that reservation into deeper children even when the immediate child count was estimated correctly. The values remain correct, but empty or short selections retain unnecessary buffer capacity. ## What changes are included in this PR? Keep Arrow take and inspect the output buffers directly through typed array accessors. Check List/LargeList offsets, Map entries, Struct fields, FixedSizeList children, validity buffers, and supported primitive/string/binary leaf storage recursively. When output storage exceeds four times its used size plus a 64-byte alignment allowance, consume the take result and compact only oversized owned buffers. Preserve ordinary capacity in other buffers, including nearly full offsets. ArrayData is constructed only for results that need compaction. This avoids constructing temporary ArrayData and does not reproduce Arrow's internal reservation formulas. The shallow empty/short selection regressions enforce retention bounds against an independent row-wise gather. In an 8,192-row three-level array fixture with equal middle lengths, the output now retains 4,227,140 bytes instead of 8,421,444 bytes. Its empty integer leaf retains zero bytes instead of 4,194,304 bytes. Logical results match an independent row-wise gather. The reconstruction preserves types, lengths, offsets, buffer contents, and validity from a valid Arrow take result. Its documented build_unchecked call avoids revalidating every nested offset or string after a capacity-only change. Buffer shrinking respects shared and external ownership. These are retained Arrow buffer capacities. Initial take allocations still occur before compaction. No RSS, peak-allocation, spill-count, or query-latency improvement is claimed. Primitive/string top-level extraction and non-NULL defaults keep their existing paths. ## How are these changes tested? - Native array tests: cargo test --offline --locked --release -p datafusion-comet-spark-expr --lib array_funcs: **96 passed**, no failures or ignored tests. Covers heterogeneous nested lengths, nullable parents/children, absent/NULL defaults, sliced inputs, and zero-/one-based, negative, NULL, and out-of-bounds ordinals. The new deep regression checks equal middle lengths with empty/short primitive grandchildren and nullable parents. - Red control on unmodified main for the shallow regression: **5 passed, 1 failed**. A separate production-expression probe against the prior compaction implementation also fails the empty-leaf capacity assertion, retaining 4 MiB. The final production expression passes that assertion, logical equality against the row-wise gather reference, and Arrow validate_full(). - Root-reactor Spark 4.1.3/JDK 21 SQL test: **1 passed**, no failed/canceled/ignored tests and no aborted suites. All eight queries assert native execution and compare with Spark, including array-of-array-of-array and array-of-map-of-array. The initial attempt aborted during socket binding; setting java.net.preferIPv4Stack=true and Spark driver host/bindAddress=127.0.0.1 allowed the suite to complete. The native DLL SHA-256 before and after the run is 8fbb7957ee3b52d91b3181cdb70568e3836f28ca4b0afcd682776bbdf6fc50ac. - Release native build and full-workspace/all-target Clippy with -D warnings passed. Cargo formatting, Maven formatting checks, Apache RAT (zero unapproved/unknown licenses), and git diff --check passed. Final Criterion verification covers 70 cases against unmodified production source at c7aa38bd0b1e7dde39cf650da43a4452a7a5ea4c. All ten initially flagged cases were independently resampled against a fresh baseline, alongside two deep empty controls. The initial regressions did not reproduce: changes for those ten cases were -13.07% to +0.53% in the second pair. Both rounds are reported below; this is a component benchmark and no query-latency speedup is claimed. The broader Spark SQL CI label `run-spark-4.1-tests` remains to be applied after publication, before queueing this native-expression change. The local Spark result above uses Spark 4.1.3. | Case | Full baseline µs | Full candidate µs | Full change | Fresh paired change | Paired 95% CI | |---|---:|---:|---:|---:|---| | list_extract_deep_nested/fixed-list/0-selected/0%-nulls | 1258.60 | 1248.92 | -0.77% | — | — | | list_extract_deep_nested/fixed-list/0-selected/25%-nulls | 1071.79 | 1079.60 | +0.73% | — | — | | list_extract_deep_nested/fixed-list/0-selected/75%-nulls | 429.27 | 424.20 | -1.18% | — | — | | list_extract_deep_nested/fixed-list/1-selected/0%-nulls | 1634.35 | 1653.46 | +1.17% | — | — | | list_extract_deep_nested/fixed-list/1-selected/25%-nulls | 1360.51 | 1420.26 | +4.39% | +0.53% | [+0.32%, +0.70%] | | list_extract_deep_nested/fixed-list/1-selected/75%-nulls | 525.85 | 543.26 | +3.31% | -0.16% | [-0.44%, +0.14%] | | list_extract_deep_nested/fixed-list/16-selected/0%-nulls | 1867.37 | 1886.91 | +1.05% | — | — | | list_extract_deep_nested/fixed-list/16-selected/25%-nulls | 1687.32 | 1692.40 | +0.30% | — | — | | list_extract_deep_nested/fixed-list/16-selected/75%-nulls | 635.90 | 649.29 | +2.11% | — | — | | list_extract_deep_nested/list-list/0-selected/0%-nulls | 267.21 | 266.74 | -0.18% | -0.96% | [-1.27%, -0.70%] | | list_extract_deep_nested/list-list/0-selected/25%-nulls | 213.22 | 212.26 | -0.45% | — | — | | list_extract_deep_nested/list-list/0-selected/75%-nulls | 89.95 | 88.14 | -2.01% | — | — | | list_extract_deep_nested/list-list/1-selected/0%-nulls | 286.58 | 284.06 | -0.88% | — | — | | list_extract_deep_nested/list-list/1-selected/25%-nulls | 233.52 | 234.81 | +0.55% | — | — | | list_extract_deep_nested/list-list/1-selected/75%-nulls | 96.84 | 94.47 | -2.45% | — | — | | list_extract_deep_nested/list-list/16-selected/0%-nulls | 1677.01 | 1729.77 | +3.15% | — | — | | list_extract_deep_nested/list-list/16-selected/25%-nulls | 1479.31 | 1479.54 | +0.02% | — | — | | list_extract_deep_nested/list-list/16-selected/75%-nulls | 401.13 | 353.05 | -11.99% | — | — | | list_extract_deep_nested/map-list/0-selected/0%-nulls | 353.84 | 351.17 | -0.75% | -0.86% | [-1.14%, -0.59%] | | list_extract_deep_nested/map-list/0-selected/25%-nulls | 290.22 | 292.00 | +0.61% | — | — | | list_extract_deep_nested/map-list/0-selected/75%-nulls | 114.06 | 113.41 | -0.57% | — | — | | list_extract_deep_nested/map-list/1-selected/0%-nulls | 360.70 | 363.30 | +0.72% | — | — | | list_extract_deep_nested/map-list/1-selected/25%-nulls | 311.81 | 312.53 | +0.23% | — | — | | list_extract_deep_nested/map-list/1-selected/75%-nulls | 117.88 | 115.33 | -2.16% | — | — | | list_extract_deep_nested/map-list/16-selected/0%-nulls | 1744.36 | 1722.75 | -1.24% | — | — | | list_extract_deep_nested/map-list/16-selected/25%-nulls | 1512.62 | 1632.36 | +7.92% | -4.93% | [-6.22%, -3.70%] | | list_extract_deep_nested/map-list/16-selected/75%-nulls | 458.38 | 517.72 | +12.95% | -13.07% | [-15.65%, -10.46%] | | list_extract_defaults/int32/no-default/0%-oob | 26.51 | 24.31 | -8.30% | — | — | | list_extract_defaults/int32/no-default/25%-null-lists-25%-null-ordinals | 28.23 | 29.82 | +5.65% | +0.08% | [-0.33%, +0.53%] | | list_extract_defaults/int32/no-default/50%-oob | 37.66 | 37.01 | -1.74% | — | — | | list_extract_defaults/int32/non-null-default/0%-oob | 65.95 | 65.90 | -0.07% | — | — | | list_extract_defaults/int32/non-null-default/25%-null-lists-25%-null-ordinals | 68.06 | 66.41 | -2.42% | — | — | | list_extract_defaults/int32/non-null-default/50%-oob | 67.82 | 68.36 | +0.79% | — | — | | list_extract_defaults/int32/null-default/0%-oob | 26.42 | 24.17 | -8.53% | — | — | | list_extract_defaults/int32/null-default/50%-oob | 37.75 | 36.95 | -2.10% | — | — | | list_extract_defaults/utf8/no-default/0%-oob | 55.32 | 52.97 | -4.25% | — | — | | list_extract_defaults/utf8/no-default/25%-null-lists-25%-null-ordinals | 47.91 | 48.75 | +1.76% | +0.44% | [+0.00%, +0.88%] | | list_extract_defaults/utf8/no-default/50%-oob | 57.44 | 57.11 | -0.57% | — | — | | list_extract_defaults/utf8/non-null-default/0%-oob | 92.26 | 92.27 | +0.01% | — | — | | list_extract_defaults/utf8/non-null-default/25%-null-lists-25%-null-ordinals | 70.26 | 68.39 | -2.66% | — | — | | list_extract_defaults/utf8/non-null-default/50%-oob | 88.56 | 87.28 | -1.45% | — | — | | list_extract_defaults/utf8/null-default/0%-oob | 55.43 | 52.96 | -4.46% | — | — | | list_extract_defaults/utf8/null-default/50%-oob | 58.27 | 56.68 | -2.73% | — | — | | list_extract_nested/list/0-selected/0%-nulls | 60.91 | 60.31 | -0.99% | — | — | | list_extract_nested/list/0-selected/25%-nulls | 57.09 | 57.75 | +1.16% | — | — | | list_extract_nested/list/0-selected/75%-nulls | 37.46 | 34.88 | -6.88% | — | — | | list_extract_nested/list/1-selected/0%-nulls | 72.57 | 72.69 | +0.17% | — | — | | list_extract_nested/list/1-selected/25%-nulls | 65.75 | 66.44 | +1.05% | — | — | | list_extract_nested/list/1-selected/75%-nulls | 40.85 | 38.96 | -4.62% | — | — | | list_extract_nested/list/128-selected/0%-nulls | 166.01 | 166.74 | +0.44% | — | — | | list_extract_nested/list/128-selected/25%-nulls | 167.50 | 166.63 | -0.51% | — | — | | list_extract_nested/list/128-selected/75%-nulls | 71.45 | 69.05 | -3.35% | — | — | | list_extract_nested/map/0-selected/0%-nulls | 131.54 | 130.91 | -0.48% | — | — | | list_extract_nested/map/0-selected/25%-nulls | 113.52 | 111.08 | -2.15% | — | — | | list_extract_nested/map/0-selected/75%-nulls | 57.22 | 55.80 | -2.48% | — | — | | list_extract_nested/map/1-selected/0%-nulls | 155.56 | 156.78 | +0.78% | — | — | | list_extract_nested/map/1-selected/25%-nulls | 128.50 | 133.58 | +3.95% | +0.49% | [+0.20%, +0.77%] | | list_extract_nested/map/1-selected/75%-nulls | 64.01 | 61.50 | -3.91% | — | — | | list_extract_nested/map/128-selected/0%-nulls | 355.48 | 366.59 | +3.13% | +0.30% | [-0.16%, +0.78%] | | list_extract_nested/map/128-selected/25%-nulls | 292.32 | 303.97 | +3.98% | -1.04% | [-2.02%, -0.26%] | | list_extract_nested/map/128-selected/75%-nulls | 118.38 | 119.41 | +0.86% | — | — | | list_extract_nested/struct-list/0-selected/0%-nulls | 64.42 | 64.18 | -0.37% | — | — | | list_extract_nested/struct-list/0-selected/25%-nulls | 62.43 | 63.59 | +1.86% | -0.96% | [-1.22%, -0.72%] | | list_extract_nested/struct-list/0-selected/75%-nulls | 42.17 | 39.92 | -5.32% | — | — | | list_extract_nested/struct-list/1-selected/0%-nulls | 82.56 | 75.82 | -8.17% | — | — | | list_extract_nested/struct-list/1-selected/25%-nulls | 71.68 | 71.78 | +0.14% | — | — | | list_extract_nested/struct-list/1-selected/75%-nulls | 45.24 | 42.86 | -5.25% | — | — | | list_extract_nested/struct-list/128-selected/0%-nulls | 169.93 | 169.02 | -0.53% | — | — | | list_extract_nested/struct-list/128-selected/25%-nulls | 173.74 | 173.10 | -0.37% | — | — | | list_extract_nested/struct-list/128-selected/75%-nulls | 76.58 | 74.17 | -3.14% | — | — | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
