> > Thinking about compatibility with existing software, RLE could possibly > even made an Extension Type that follows the layout of a struct of > int32 and the encoded value type. I'm wondering wether this would be > better for compatibility.
I might be misunderstanding this proposal, but I don't think this works. Wouldn't the structs with RLE have different row lengths then any Array in the same record/batch and table? I think this means that validation would fail on them. On Fri, Jun 3, 2022 at 8:22 AM Tobias Zagorni <tob...@zagorni.eu.invalid> wrote: > > Well, Arrow C++ does not have a notion of encoding distinct from the > > data type. Adding such a notion would risk breaking compatibility for > > all existing software that hasn't been upgraded to dispatch based on > > encoding. > > Thinking about compatibility with existing software, RLE could possibly > even made an Extension Type that follows the layout of a struct of > int32 and the encoded value type. I'm wondering wether this would be > better for compatibility. > > The format is very similar to what was proposed before, exept that the > indices now become an own child array, and there is in additional > validity bitmap in the parent that could just be NULLPTR. > Another adventage here would be that it is obvous how to add support > for other run-length types than int32 if needed. > > Best, > Tobias >
