Le 31/05/2022 à 21:41, Micah Kornfield a écrit :
I'm currently working on adding Run-Length encoding to arrow.
Nice
What are the intended use cases for this:
- external engines want to provide run-length encoded data to work on
using arrow?
It is more than just external engines. Many popular file formats support
RLE encoding. Being able to natively transfer this into arrow memory
without exploding it saves CPU time. Similarly, for things like flight it
can dramatically decreases data on the wire
For things like Flight (or more generally the IPC format) you can also
enable LZ4 compression (which costs CPU time, but will probably save
space much more efficiently than a crude RLE scheme with 32-bit offsets).
Regards
Antoine.
