Hi,
I am trying to convert pyarrow buffers into Rust buffers and vice-versa, to
perform zero-copy from and to pyarrow, to and from Rust's library.
I was able to perform the operation rust -> pyarrow, using something along
the lines of
// 64 bits system
let pointer = buffer.raw_data() as i64;
pyarrow.call1("foreign_buffer
<https://arrow.apache.org/docs/python/generated/pyarrow.foreign_buffer.html>",
(pointer, buffer.len()))
However, I am not being able to perform the opposite transformation, due to
a memory misalignment check that is failing. I.e.
let address = buffer.getattr("address")?.extract::<i64>()?;
let size = buffer.getattr("size")?.extract::<i64>()? as usize;
// panics with "memory not aligned"
Buffer::from_raw_parts(address as *const u8, size, size)
I get an address such as `4624199872` (i64), but, when converted to `*const
u8`, our rust implementation does not consider it to be memory aligned (at
least in x86_64) and panics. Has anyone worked in this problem? Any hints
on what I am doing wrong?
AFAIK, our rust implementation requires different alignments, depending on
the architecture (see memory.rs
<https://github.com/apache/arrow/blob/master/rust/arrow/src/memory.rs>).
Wasn't this supposed to be a single alignment, consistent throughout the
arrow? I am confused.
Any help would be greatly appreciated!
Best,
Jorges
