The email chain concerns IPC, in particular that arrow-cpp's IPC reader can
return unaligned buffers. Unless compiled with sanitizers, this will not
visibly cause issues so long as the data never leaves arrow-cpp, however, when
data is sent over FFI other arrow implementations, including arrow-r
First, the subject of this email is IPC, which is confusing. From the
discussion it sounds like we are primarily talking about FFI. It sounds
like the options here are:
1. Silently realign unless user opts out of realignment
The user will not get any runtime errors (easier to use) but a method
If that is the eventual outcome of this discussion, I would be happy to,
however, I'd like to avoid fragmenting the discussion just yet.
On 27/03/2025 17:18, Antoine Pitrou wrote:
Le 27/03/2025 à 18:14, Raphael Taylor-Davies a écrit :
It's obviously preferrable to be zero-copy but it's certai
Le 27/03/2025 à 18:14, Raphael Taylor-Davies a écrit :
It's obviously preferrable to be zero-copy but it's certainly not
mandatory, especially as the data being shared is assumed to be
read-only in most use cases.
In which case we should probably remove the comment about alignment from
the C i
It's obviously preferrable to be zero-copy but it's certainly not
mandatory, especially as the data being shared is assumed to be
read-only in most use cases.
In which case we should probably remove the comment about alignment from
the C interface specification, and highlight that implementatio
Hello,
Le 27/03/2025 à 17:53, Raphael Taylor-Davies a écrit :
The current ambiguity, however, makes it hard to set reasonable
defaults, as it isn't clear if FFI should be zero-copy and therefore
have alignment restrictions or not.
It's obviously preferrable to be zero-copy but it's certainl
Hi All,
Further context on Felipe's comment can be found [1]. You will note
arrow-rs can align on read, and in fact does do so by default in its IPC
reader, and by extension its flight client, the objection was solely to
always doing this over FFI, where it should at the very least be
configu
Hi,
All this complexity everywhere when arrow-rs could simply check the
alignment when they ingest external buffers and re-allocate to ensure
alignment.
I'm in favor of producers of Arrow arrays like a Flight client ensuring
alignment as early as possible (when buffers are allocated for arrays
de
Hi devs,
we are planning to add a new IPC read option:
|IpcReadOptions.ensure_memory_alignment|
This makes the IPC reader ensure that returned data are memory aligned
(as for instance required by Apache Arrow Rust).
Default is |false| as it is understood that data from gRPC always has to
be re
Hi,
This seems like a sensible approach and an improvement to developer/user
experience.
Rusty
10 matches
Mail list logo
