Hi Gabor Thanks for creating this discussion thread. This is indeed a good topic to discuss.
The idea was to have lightweight compression for the footer for cass when Puffin files are bigger. It is true that the implementation didn't follow the spec yet. If we remove this from the Puffin spec, we will probably want to add it later. The Iceberg implementation was supposed to be based on aircompressor pure Java implementation https://github.com/airlift/aircompressor/pull/142. AFAICT, aircompressor started to favor (or be more OK with) native implementations (because of Project Panama), so adding LZ4 framed compression might be simpler these days. I would prefer to spend the effort on completing the compression. Best Piotr On Tue, 27 Aug 2024 at 14:29, Gabor Kaszab <gaborkas...@cloudera.com.invalid> wrote: > Hi Iceberg Community, > > I saw in the Puffin spec <https://iceberg.apache.org/puffin-spec> that > the footer of the Puffin file or the blobs themselves could be compressed > by LZ4. I checked the code > <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/puffin/PuffinFormat.java#L110> > however, and for me it seems that currently LZ4 is not supported. > My first question is do I miss anything here? > The second, is if we in fact don't support LZ4, can I remove it from the > spec to avoid confusions? (I believe this requires a vote in a separate > thread) > > Thanks, > Gabor > >
