Yes On Tue, Jun 11, 2019 at 9:16 AM Wes McKinney <wesmck...@gmail.com> wrote:
> Sounds reasonable. John, would you like to create a C++ implementation > of this? It would be helpful for a vote to have an initial patch. > > On Mon, Jun 10, 2019 at 12:24 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > Sorry for the late reply. I think it sounds reasonable to have custom > > metadata in the footer as well. > > > > Even though the change isn't likely to have a large impact, it still > might > > pay to have a vote, since it potentially could require (small) code and > > integration test updates to ensure the data is round-tripped correctly. > > > > On Thu, May 30, 2019 at 11:38 AM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > hi John, > > > > > > In principle I don't see a problem with adding a custom_metadata: > > > [KeyValue] field to the Footer table in File.fbs > > > > > > https://github.com/apache/arrow/blob/master/format/File.fbs#L26 > > > > > > This field could be used for application-level information about the > > > file's provenance, etc. > > > > > > Note here are the other places where we have such fields: > > > > > > * Field > > > * Schema > > > * Message > > > > > > An alternative solution would be to handle such metadata in a separate > > > file, but I see the benefit of encapsulation in a single file. > > > > > > I'm not sure this is a serious enough change to require a vote but I > > > will be interested in the opinions of others. > > > > > > - Wes > > > > > > On Wed, May 29, 2019 at 7:52 AM John Muehlhausen <j...@jgm.org> wrote: > > > > > > > > Original write of File: > > > > > > > > Schema: custom_metadata: {"value":1} > > > > Message > > > > Message > > > > Footer > > > > Schema: custom_metadata: {"value":1} > > > > > > > > > > > > Process appends messages (new data in bold): > > > > > > > > Schema: custom_metadata: {"value":1} > > > > Message > > > > Message > > > > *Message* > > > > *Footer* > > > > * Schema: custom_metadata: {"value":2}* > > > > > > > > > > > > Re-writing the entire file may not be practical due to the size, so > after > > > > this operation custom_metadata at the Schema level is out of sync... > the > > > > two Schema copies are no longer the same. > > > > > > > > In order to address this I would like to propose adding > custom_metadata > > > to > > > > Footer as well. Any objections? > > > > > > > > -John > > > >
