We have protocol-level compression for message body buffers [1][2] with LZ4 or ZSTD
In-memory compression and encoding other than dictionary encoding (like RLE) has been discussed multiple times and remains on the roadmap for the project. [1]: https://github.com/apache/arrow/blob/master/format/Message.fbs#L45 On Tue, Sep 15, 2020 at 7:18 PM Jacob Quinn <quinn.jac...@gmail.com> wrote: > > Am I correct in understanding there's nothing in the arrow ipc/file format > spec about compression? I thought I had seen something at one point, but > looking over the spec website, I don't see anything. > > -Jacob
