Re: Alignment not stored in arrow metadata

2021-04-07 Thread Micah Kornfield
> > AFAIK IPC is just bytes. The alignment is done when they are copied over to > allocated memory regions. Agreed, that if implementations are copying then this isn't a concern. The IPC and File Formats were designed for memory mapping/zero copy. So there is an assumption that kernel pages mee

Re: Alignment not stored in arrow metadata

2021-04-07 Thread Jorge Cardoso Leitão
Hi Jacob, AFAIK IPC is just bytes. The alignment is done when they are copied over to allocated memory regions. It is the implementations' responsibility to allocate memory regions that are aligned depending on how those bytes should be interpreted (e.g. u64 vs u8). This interpretation is induced

Alignment not stored in arrow metadata

2021-04-06 Thread Jacob Quinn
As far as I can tell, the alignment padding used in an IPC stream/file isn't stored explicitly, and not really "inferrable", though maybe technically possible if you calculated what bytes are *necessary* given a buffer's data vs. what's actually stored. Just wondering if this has been brought up a