Sounds reasonable. John, would you like to create a C++ implementation of this? It would be helpful for a vote to have an initial patch.
On Mon, Jun 10, 2019 at 12:24 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > > Sorry for the late reply. I think it sounds reasonable to have custom > metadata in the footer as well. > > Even though the change isn't likely to have a large impact, it still might > pay to have a vote, since it potentially could require (small) code and > integration test updates to ensure the data is round-tripped correctly. > > On Thu, May 30, 2019 at 11:38 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > hi John, > > > > In principle I don't see a problem with adding a custom_metadata: > > [KeyValue] field to the Footer table in File.fbs > > > > https://github.com/apache/arrow/blob/master/format/File.fbs#L26 > > > > This field could be used for application-level information about the > > file's provenance, etc. > > > > Note here are the other places where we have such fields: > > > > * Field > > * Schema > > * Message > > > > An alternative solution would be to handle such metadata in a separate > > file, but I see the benefit of encapsulation in a single file. > > > > I'm not sure this is a serious enough change to require a vote but I > > will be interested in the opinions of others. > > > > - Wes > > > > On Wed, May 29, 2019 at 7:52 AM John Muehlhausen <j...@jgm.org> wrote: > > > > > > Original write of File: > > > > > > Schema: custom_metadata: {"value":1} > > > Message > > > Message > > > Footer > > > Schema: custom_metadata: {"value":1} > > > > > > > > > Process appends messages (new data in bold): > > > > > > Schema: custom_metadata: {"value":1} > > > Message > > > Message > > > *Message* > > > *Footer* > > > * Schema: custom_metadata: {"value":2}* > > > > > > > > > Re-writing the entire file may not be practical due to the size, so after > > > this operation custom_metadata at the Schema level is out of sync... the > > > two Schema copies are no longer the same. > > > > > > In order to address this I would like to propose adding custom_metadata > > to > > > Footer as well. Any objections? > > > > > > -John > >
