kronic opened a new issue, #26909:
URL: https://github.com/apache/datafusion/issues/26909
### Is your feature request related to a problem or challenge?
DataFusion is frequently embedded as an in-process engine that serves data
to other processes: the engine runs in a dedicated "database" process, while
consumers read results over the Arrow C Data Interface (directly, or through an
ADBC driver or a custom FFI layer).
That path cannot be zero-copy today, and the failure mode is hard to
diagnose.
The C Data Interface is pointer-based. ArrowSchema/ArrowArray handed to a
consumer contain absolute addresses (format, children, buffers, …) and a
release callback. DataFusion produces those buffers on its own heap, so:
- a consumer in another process cannot dereference buffers[i], children[i],
format — the addresses are not mapped there, which results in an access
violation or, worse, in silently dangling pointers that only fail when the
first value is read;
- the release callback is a function pointer into the producer's code, so
the consumer must not call it either.
As a result, the only correct way to move a result set across a process
boundary is to copy every buffer, or to serialize the data (Arrow IPC). Both
cost a full pass over the data, which is exactly what users of an embedded
engine are trying to avoid.
DataFusion currently offers no way to influence where result buffers live:
- MemoryPool performs accounting and spilling only; it does not select an
address space.
- RuntimeEnv/SessionConfig have no allocator hook.
The allocation itself happens in arrow-rs MutableBuffer, which goes through
the process-global allocator and cannot be redirected afterwards.
So "run DataFusion in one process and read its results in another without
copying" is not expressible with the current API, even though DataFusion's own
execution model (partitioned, streaming, RecordBatch-based) would fit such a
deployment well.
### Describe the solution you'd like
An opt-in way to have a query's output buffers allocated in a
caller-provided memory region, plus a descriptor that lets a consumer
reconstruct the batches in its own address space.
Sketch of the API surface:
```rust
// Region is created by the application: named shared memory / memory-mapped
file.
let region = SharedRegion::create("df_results", 1 << 30)?;
let config = SessionConfig::new()
.with_buffer_allocator(region.allocator()); // new
let df = ctx.sql("SELECT ... FROM parquet_scan(...)").await?;
let mut stream = df.execute_stream().await?;
while let Some(batch) = stream.next().await {
// Batch buffers live inside `region`; publish() returns only
offsets/lengths,
// so the consumer never needs producer addresses.
let descriptor = region.publish(batch?)?;
send_descriptor_to_consumer(descriptor)?;
}
```
Two properties matter:
Allocator threading. A BufferAllocator handle carried by
TaskContext/RuntimeEnv and used at buffer creation sites, defaulting to the
current global allocator so behavior is unchanged unless opted in. Because
operators run on multiple threads and also spawn work (repartition, joins,
parquet prefetch), a thread-local set at the call site is not sufficient — the
allocator has to travel with the task.
Descriptor-based export. The consumer should receive (region handle, buffer
offsets, lengths, offsets/validity layout) rather than absolute pointers, then
build its own ArrowArray with buffers[i] = local_base + offset. This avoids the
requirement that the region be mapped at the same base address in both
processes, which is fragile in practice (address-space collisions, ASLR,
needing VirtualAlloc(MEM_RESERVE) / mmap(MAP_FIXED_NOREPLACE)).
With that, the data is written once into shared memory, the consumer maps
the region and reads zero-copy, and release stays a local no-op on the consumer
side.
### Describe alternatives you've considered
- Deep copy on the consumer side. Walk the schema/array graph and copy every
buffer into the consumer's memory (e.g. ReadProcessMemory on Windows,
process_vm_readv on Linux). Works, but it is a full memcpy of the result set —
the same cost as IPC, without the portability.
- Arrow IPC. Standard and supported, but serializes the data and adds
framing/deserialization overhead.
- A custom driver/FFI layer that copies each batch once into a memory-mapped
region. This is workable today and is essentially the feature requested here,
just implemented outside DataFusion — it keeps DataFusion untouched but
duplicates the logic in every embedder.
- Replacing the global allocator in the host process. A custom
#[global_allocator] that routes allocations with Arrow's buffer alignment
(ALIGNMENT = 128 on x86_64) into a shared arena. It affects the whole process
(tokio, parsers, catalog), misses buffers created from Vec/bytes::Bytes, and
depends on an implementation detail of arrow-rs rather than on a contract.
- Patching arrow-rs. Adding a per-buffer allocator to arrow-buffer and
threading it through the execution context. Correct in principle, but it means
maintaining a fork of arrow-rs and DataFusion and pinning versions.
- ADBC partitions / ConnectionReadPartition. Designed to be passable across
processes, but the partition payload is serialized, so it is again a copy.
### Additional context
- The allocation site is arrow-buffer's MutableBuffer → std::alloc with
Layout { align: 128 }. A hook in DataFusion can only be effective if it reaches
those allocation sites; that is why the "allocator travels with the task" part
is the essential piece, not just a config flag.
- A partial solution is still useful: even if some buffers (e.g. those built
from Vec by the parquet reader) end up outside the region, exposing the
mechanism plus a per-batch "is this batch fully region-backed?" flag would let
callers fall back to a copy only when necessary.
- Precedent for the descriptor approach: the Arrow C Device Data Interface
separates the device representation from the arrays themselves
(ArrowDeviceArray + ArrowMemoryManager). A host shared-memory device type there
would be the natural long-term home for this, but DataFusion-side support for
placing batches in a region is a prerequisite.
- This is an API/design request, not a crash report; the current behavior is
expected given a private heap, but it is undocumented and easy to hit when
embedding DataFusion as a separate service process.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
