alamb opened a new issue, #26936: URL: https://github.com/apache/datafusion/issues/26936
### Is your feature request related to a problem or challenge? Parquet 2.14.0 added the ALP encoding for `FLOAT` / `DOUBLE` columns (see the [ALP blog post](https://parquet.apache.org/blog/2026/09/22/alp-adaptive-lossless-floating-point-encoding-in-apache-parquet/)), and the `parquet` crate reads and writes it since arrow-rs 60.0.0 (apache/arrow-rs#9372). DataFusion `main` is on arrow/parquet 60, so it can already *read* ALP encoded files, but there is no way to *write* them from SQL or the config: ```sql > COPY (SELECT round(68 + 15 * sin(x / 5000.0) + random(), 2) AS temp_f FROM generate_series(1, 1000000) AS t(x)) TO 'readings_alp.parquet' OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' 'false'); Error: Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: alp. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split. ``` The same happens with `SET datafusion.execution.parquet.encoding = 'alp'`. The cause is that `DFParquetEncoding` in `datafusion/common/src/parquet_config.rs` has no `Alp` variant, even though `parquet::basic::Encoding::ALP` exists and `"alp".parse::<Encoding>()` works. For reference, writing the same 1M two-decimal doubles with `parquet-rewrite --encoding alp` from arrow-rs produces a file that DataFusion reads fine and that is 42% smaller than `PLAIN` with the same `ZSTD(1)` compression (1,088,502 vs 1,882,294 bytes). ### Describe the solution you'd like 1. Add an `Alp` variant to `DFParquetEncoding` with the `FromStr` / `Display` / `From<DFParquetEncoding> for parquet::basic::Encoding` plumbing, so `'alp'` is accepted by `datafusion.execution.parquet.encoding` and the per-column `format.encoding` options. 2. Mention it in the config docs (`docs/source/user-guide/configs.md`) and the parquet writer option docs. 3. Add a sqllogictest that writes a `DOUBLE` column with `'format.encoding' 'alp'` and checks `parquet_metadata(...)` reports `ALP` in `encodings`. Note that ALP only applies to `FLOAT` / `DOUBLE`; the parquet writer falls back for other types the same way it does for `BYTE_STREAM_SPLIT`, so no extra validation should be needed in DataFusion. ### Describe alternatives you've considered Writing with the `parquet` crate directly (as above) works today, but users of `COPY ... TO` and `datafusion-cli` cannot opt in. ### Additional context - Parquet spec: https://github.com/apache/parquet-format/blob/master/Encodings.md#ALP (parquet-format 2.14.0, "Preview") - arrow-rs implementation: apache/arrow-rs#9372; tracking issue apache/arrow-rs#10541 - Related: `BYTE_STREAM_SPLIT` was added to `DFParquetEncoding` the same way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
