nat-openai opened a new pull request, #6948: URL: https://github.com/apache/datafusion-comet/pull/6948
## Which issue does this PR close? Addresses part 2 of [#5712](https://github.com/apache/datafusion-comet/issues/5712): memory accounting for integral `sequence` construction. ## Rationale for this change Integral `sequence` can allocate a large Arrow values buffer without consulting Comet’s memory pool. Fallible allocation handles allocator refusals, but does not enforce the task’s memory budget. This change reserves construction memory through the task’s existing pool before allocating buffers. The reservation lasts only for the invocation; downstream buffering operators remain responsible for accounting for the output they retain. ## What changes are included in this PR? - Construct the sequence UDF through the scalar-function factory using `SessionState`, preserving the session’s exact memory pool and tracking wrappers. - Use staged `try_grow` reservations for column inputs: reserve offsets, validity buffers and length scratch first, then reserve values after validating and caching each row’s length once. - Retain a single reservation request for scalar, empty and metadata-proven all-null inputs. Construct complete scalar batches within the reservation so native constant folding cannot bypass admission. - Release reservations and consumer registrations on return, errors and unwinding. Preserve Spark’s null handling, overflow checks and error precedence, while propagating non-resource pool errors unchanged. - Add pooled benchmarks alongside the existing kernel benchmarks and document the reservation’s scope and limitations. The check follows the existing pool policy and introduces no new memory setting. A refusal can occur before a downstream native operator reaches its spill step; this does not add native reclamation or guarantee a process-memory ceiling. On-heap mode retains its existing unbounded pool policy. ## How are these changes tested? - Native tests cover both reservation stages, refusal and recovery, concurrent admission, scalar expansion, null and sliced inputs, error precedence, interruption propagation, and cleanup while returned arrays or FFI exports remain alive. - Factory and planner tests verify session-pool wiring. Spark tests cover bounded-pool refusal, recovery with smaller batches, and sorting and collecting sequence output. - The new `sequence_pooled` Criterion group exercises the UDF with a tracked, bounded pool across short, long, nullable and scalar inputs. Preliminary diagnostics retain approximately **5–7% overhead** for the two-element column/null cases and **12.5% for default singleton rows**; performance parity is not claimed. Validation against the pinned **DataFusion 55.2**, JVM test execution and production JNI performance measurements remain pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
