hi David -- would you like to open a PR and corresponding JIRA issue for discussion? We might want to hold a vote to formalize the extension type mechanism (and to fix the metadata names -- I agree that having an ARROW namespace would be better than what we are doing now)
On Thu, May 2, 2019 at 7:02 AM David Li <li.david...@gmail.com> wrote: > > Re: Java support, I've sketched out an implementation: > https://github.com/lihalite/arrow/pull/2 > > On 5/1/19, Micah Kornfield <emkornfi...@gmail.com> wrote: > >> > >> I'm awaiting community feedback about the approach to implementing > >> extension types, whether the approach that I've used (using the > >> following keys in custom_metadata [1]) is the one that we want to use > >> longer-term. This certainly seems like a good time to have that > >> discussion. If there is consensus then we can document it formally in > >> the specification documents, and we probably will want to hold a vote > >> to ensure that we are in agreement. > >> > > > > Please let me know if this is best on a separate thread. I think I would > > feel more comfortable finalizing this if we had a few more examples > > exercising the functionality. Inet, seems like a complicated enough > > use-case for modeling which would make it a good use-case (It seems like it > > might involve a struct/union?). I also presume we will need a Java > > implementation, before we finalize anything? > > > > A small amount of bikeshedding on key names: We should probably take a > > namespace reservation approach for custom metadata in Schema.fbs [1]. In > > this regard I have a small preference for something reserving all metadata > > with something like "ARROW:<reserved_arrow_key>" or "ARROW." (not an > > underscore, and I'm open to different capitalization.) This seems to be a > > similar approach to how avro reserves metadata keys [2]. > > > > [1] > > https://github.com/apache/arrow/blob/b8aeb79e94a5a507aeec55d0b6c6bf5d7f0100b2/format/Schema.fbs#L264 > > [2] https://avro.apache.org/docs/1.8.1/spec.html > >
