Le 20/10/2020 à 12:09, Xie, Qi a écrit :
> Hi, Wes
>
> Yes currently the purpose of the key-value metadata is just a hint to
> indicate that the parquet file is compressed by plugin so that the parquet
> reader can load the plugin library and use plugin to decompress the file.
> There are many optimized GZIP implementations and may not compatible with the
> standard gzip, for example due to hardware limit, the HW-GZIP history window
> size maybe smaller than the standard gzip, so that HW-GZIP can't decompress
> the file compressed by standard gzip and because we are still use the
> Compression::GZIP as Compression::type, we need that metadata to distinguish
> it from the standard gzip.
What does it bring over ZSTD or LZ4 exactly?
Regards
Antoine.
