[
https://issues.apache.org/jira/browse/ARROW-384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690537#comment-15690537
]
Wes McKinney commented on ARROW-384:
------------------------------------
I talked this proposal through with Julien yesterday and we should be able to
manage without additional metadata by treating the contiguous buffer block
written as part of the file format as a "memory page" for the purposes of
record batch reconstruction -- so we will not consider the offset from the
beginning of the physical file to be semantically meaningful. This is distinct
from the shared memory IPC case where the buffers may occur in many different
places in one or more shared memory regions.
I'll write up some details under format/ about this more precisely in the
course of implementing this in my patch for ARROW-363.
> Align Java and C++ RecordBatch data and metadata layout
> -------------------------------------------------------
>
> Key: ARROW-384
> URL: https://issues.apache.org/jira/browse/ARROW-384
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Julien Le Dem
>
> layout on C++ side:
> {noformat}
> <buffers> <metadata> <metadata size: int32>
> {noformat}
> and on the java side:
> {noformat}
> <metadata> <buffers>
> {noformat}
> In the file format the footer has a Block info that contains the metadata
> length.
> https://github.com/apache/arrow/blob/f082b17323354dc2af31f39c15c58b995ba08360/format/File.fbs#L36
> See:
> https://github.com/apache/arrow/pull/211#issuecomment-262080545
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
