Hi, Today I got the opportunity to talk with Jim Pivarski, the main developer on the OAMap project (*). Under the hood, he is doing something not unlike the Arrow representation of nested arrays: he stores and processes structured data as linear arrays, allowing very fast processing on seemingly irregular data (in Array parlance, think something like lists of lists of structs). It seems that OAMap data requires two kinds of logical types that Arrow misses :
- a pointer type, where a physical array of ints is used to represent indices into another array (the logical value being of course the value pointed to) - a span type, where two physical arrays of ints are used to represent start and stop indices into another array (the logical value being the list of values delimited by the start / stop indices) Did such a feature request already come by? Is this something we should add to our roadmap or future wishlist? (*) https://github.com/diana-hep/oamap Regards Antoine.
