voonhous opened a new issue, #20003: URL: https://github.com/apache/hudi/issues/20003
Follow-up to #18780 (RFC-105 hudi-trino migration), raised in review of trinodb/trino#30682. ### What `hudi-trino` adds a `hudi.parquet.use-column-index` config property and a `parquet_use_column_index` session property, both defaulting to `false`. The property first appeared with the migration of the connector into this repo (#13493). Trino's in-tree Hudi connector has no such property. It takes the setting from `ParquetReaderConfig`, where `parquet.use-column-index` defaults to `true`, and Trino's Hive, Iceberg and Delta Lake connectors also default `parquet_use_column_index` to `true`. So once trinodb/trino#30682 switches Trino to `hudi-trino`, standalone Hudi catalogs stop using Parquet column indexes by default, while Hudi tables read through Trino's lakehouse catalog keep using them. ### Question Is the `false` default a correctness guard (for example, a known problem with column indexes in Hudi-written files) or just a conservative choice? - If it is conservative, align the default with Trino's `true`. - If it guards against a known problem, document the reason so Trino can document the difference between Hudi and lakehouse catalogs. ### Why here Under RFC-105, `HudiConfig` and `HudiSessionProperties` live in `org.apache.hudi:hudi-trino`, so the default can only change here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
