At least for Flight, I don't think we'd use that. Right now the way compression is supported is the same way as with Feather, i.e. the body buffers in each individual record batch sent on the wire are compressed, but not the stream as a whole. (And so far we haven't found a compelling benefit for compression in Flight in general.)
Best, David On Thu, Mar 11, 2021, at 14:34, Antoine Pitrou wrote: > > Le 11/03/2021 à 19:54, Micah Kornfield a écrit : > >> > >> Indeed, I don't think it was discussed publicly. The LZ4 frame format > >> has several things going for it: > >> - it allows streaming compression and decompression (meaning you can > >> avoid loading a huge compressed buffer at once) > > > > Is this something we make use of or intend to make use of? > > Good question. Currently we don't. Perhaps David Li wants to answer > this, since he's been working a lot on Flight. > > >> - it embeds the decompressed size, allowing exact allocation of the > >> decompressed buffer > > > > IIUC, We already do this in the IPC specification (the first 8 bytes of the > > compressed buffer are used for this). > > Ah, you're right. It doesn't matter then. > > > - it has an optional checksum > > > > This seems like a good thing, so probably worth keeping (although it would > > be the only place where we do checksums today). > > (or of course we could add an optional higher-level checksum in the IPC > format) > > Regards > > Antoine. >
