Hi Jorge,
This is correct to my knowledge offsets are not modelled in IPC. There was
a lot of debate on whether to include them in the c data interface.
Cheers,
Micah
On Friday, February 12, 2021, Jorge Cardoso Leitão <jorgecarlei...@gmail.com>
wrote:
> Hi,
>
> I am going through the Rust implementation of the IPC, and I am trying to
> understand how we share Arraydata offsets.
>
> Specifically, our C data interface supports the notion of an offset,
> measured in slots, that denotes how many slots ahead of the buffer pointers
> we read from. This enables us to share buffers between arrays, as we can
> create a new array out of an existing array by slicing, and have its offset
> increased.
>
> However, I can't find this notion in our IPC format. We could argue that
> IPC format does not support that, and that we should just slice our buffers
> before encoding them to the message. That works for any type whose
> bit_width is a multiple of 8, but I can't see this to work for bitmap
> buffers: when we have an offset of 3 < 8 slots and a buffer with a validity
> bitmap, we can't slice that buffer by 3 bits. If we do not share the offset
> information and the consumer assumes an offset of 0, we will be consuming a
> buffer offsetted by 0, and we lost the 3 in communication, thereby
> incorrectly reconstructing the validity bitmap.
>
> One solution is to assume an offset of zero when reading from IPC. But afai
> understand, in that case, producers must themselves only share bitmap
> buffers that are aligned at "8 bit boundaries". For example, an array with
> offset 3, len 12 and a (shared) validity buffer with
>
> 01101010, 01101010
>
> can't just write the above to the message; it must write the "new" below:
>
> new: (010){01101}, 0000[1101]
> old: {01101}010, 0[1101](010) # 12 + 3 = 15, unbracket bits are ignored
>
> i.e. skip the first 3 bits from the first byte and shift all bits
> accordingly.
>
> Is this reasoning correct? Is this the intention?
>
> Best,
> Jorge
>
