hi Bryan, Thanks for bringing this up again. I will reply in some more detail, but to help could you create a major section in
https://cwiki.apache.org/confluence/display/ARROW/Columnar+Format+1.0+Milestone and include these details? We are falling significantly short of hardening a v1.0 iteration of the columnar format, and having a single document listing out all the work that needs to be done (including the Map type) is a good way to help herd the cats. Thanks Wes On Tue, Jul 10, 2018 at 5:30 PM, Bryan Cutler <cutl...@gmail.com> wrote: > Hello All, > > I would like to start moving forward with Map type support and begin > working on implementations. I believe we just need to define the specifics > of the metadata representation before getting started. Previously, there > was a thread [1] that discussed adding Map as a logical type and I'll try > to summarize where we are currently. > > Map has been added as a logical type and defined in the Flatbuffer schema > format with 1 field "keysSorted" which indicates if the child keys vector > has been presorted. A Map is a nested type that is represented as > List<entry: Struct<key: K, value: V>>. > > I think these are the 2 main issues of the metadata that need to be agreed > upon: > > - Same memory layout as List<entry: Struct<key: K, value: V>>. This is so > implementations lacking Map can alias as repeated struct values. > > - `Struct` and `K` fields are constrained to be non-nullable, other fields > can be nullable > > > Here is a sample JSON metadata representation: > > { > "name" : "MapName", > "nullable" : true|false, > "type" : { > "name" : "map", > "keysSorted" : true|false > }, > "children" : [{ > "name" : "entry", > "nullable" : false, > "type" : { > "name" : "struct" > }, > "children" : [{ > "name" : "key", > "nullable" : false, > "type" : { > "name" : K > }, > "children" : [] > },{ > "name" : "value", > "nullable" : true|false, > "type" : { > "name" : V > }, > "children" : [] > }] > }] > > > Any concerns or objections to the above? Hopefully that covers what needs > to be discussed, please correct me if I missed something. Thanks! > > Bryan > > > [1]: > https://lists.apache.org/thread.html/d61f21924159718fb31d27f5c85d58d393a88708f76dff510c8da322@%3Cdev.arrow.apache.org%3E
