Le 28/01/2021 à 17:59, Joris Peeters a écrit : > From Python, I'm dumping an LZ4-compressed arrow stream to a file, using > > with pa.output_stream(path, compression = 'lz4') as fh: > writer = pa.RecordBatchStreamWriter(fh, table.schema) > writer.write_table(table) > writer.close() > > I then try reading this file from Java, starting with > > var is = new LZ4FrameInputStream(new FileInputStream(path.toFile())); > > using the lz4-java library. That fails, however, with
Well, that sounds expected. LZ4 compression in the IPC format does not work by compressing the whole stream. Instead, buffers in the stream are compressed individually, while metadata is uncompressed. So, you needn't wrap the stream with LZ4 yourself. Instead, just let the Java implementation of Arrow handle compression. It *should* work. Regards Antoine.
