Sorry for the late reply. I think it sounds reasonable to have custom metadata in the footer as well.
Even though the change isn't likely to have a large impact, it still might pay to have a vote, since it potentially could require (small) code and integration test updates to ensure the data is round-tripped correctly. On Thu, May 30, 2019 at 11:38 AM Wes McKinney <wesmck...@gmail.com> wrote: > hi John, > > In principle I don't see a problem with adding a custom_metadata: > [KeyValue] field to the Footer table in File.fbs > > https://github.com/apache/arrow/blob/master/format/File.fbs#L26 > > This field could be used for application-level information about the > file's provenance, etc. > > Note here are the other places where we have such fields: > > * Field > * Schema > * Message > > An alternative solution would be to handle such metadata in a separate > file, but I see the benefit of encapsulation in a single file. > > I'm not sure this is a serious enough change to require a vote but I > will be interested in the opinions of others. > > - Wes > > On Wed, May 29, 2019 at 7:52 AM John Muehlhausen <j...@jgm.org> wrote: > > > > Original write of File: > > > > Schema: custom_metadata: {"value":1} > > Message > > Message > > Footer > > Schema: custom_metadata: {"value":1} > > > > > > Process appends messages (new data in bold): > > > > Schema: custom_metadata: {"value":1} > > Message > > Message > > *Message* > > *Footer* > > * Schema: custom_metadata: {"value":2}* > > > > > > Re-writing the entire file may not be practical due to the size, so after > > this operation custom_metadata at the Schema level is out of sync... the > > two Schema copies are no longer the same. > > > > In order to address this I would like to propose adding custom_metadata > to > > Footer as well. Any objections? > > > > -John >
