hi John, In principle I don't see a problem with adding a custom_metadata: [KeyValue] field to the Footer table in File.fbs
https://github.com/apache/arrow/blob/master/format/File.fbs#L26 This field could be used for application-level information about the file's provenance, etc. Note here are the other places where we have such fields: * Field * Schema * Message An alternative solution would be to handle such metadata in a separate file, but I see the benefit of encapsulation in a single file. I'm not sure this is a serious enough change to require a vote but I will be interested in the opinions of others. - Wes On Wed, May 29, 2019 at 7:52 AM John Muehlhausen <j...@jgm.org> wrote: > > Original write of File: > > Schema: custom_metadata: {"value":1} > Message > Message > Footer > Schema: custom_metadata: {"value":1} > > > Process appends messages (new data in bold): > > Schema: custom_metadata: {"value":1} > Message > Message > *Message* > *Footer* > * Schema: custom_metadata: {"value":2}* > > > Re-writing the entire file may not be practical due to the size, so after > this operation custom_metadata at the Schema level is out of sync... the > two Schema copies are no longer the same. > > In order to address this I would like to propose adding custom_metadata to > Footer as well. Any objections? > > -John
