Re: Join operation on attributes from arrow structs

2020-04-02 Thread Francois Saint-Jacques
They're mapped with the StructType/StructArray, which is also columnar representation, e.g. one buffer per field in the sub-object. If you have varying/incompatible types, a field will be promoted to a UnionType. François On Thu, Apr 2, 2020 at 12:54 AM Micah Kornfield wrote: > > Hi Hasara, > Th

Re: Join operation on attributes from arrow structs

2020-04-01 Thread Micah Kornfield
Hi Hasara, There isn't current functionality in C++/Python to do this ( https://issues.apache.org/jira/browse/ARROW-4630 is the issue tracking this). Also how nested attributes in json format are mapped into buffers once > converted in arrow format? I'm not sure I understand this question? Thank

Join operation on attributes from arrow structs

2020-03-22 Thread Hasara Maithree
Hi all, Assume I have a json file named 'my_data.json' as below. *{"a": [1, 2], "b": {"c": true, "d": "1991-02-03"}} {"a": [3, 4, 5], "b": {"c": false, "d": "2019-04-01"**}}* If I need to do a join operation based on attribute d, can I do it directly from arrow structs? ( or are there any effici