huaxingao commented on PR #1723: URL: https://github.com/apache/datafusion-comet/pull/1723#issuecomment-2876826410
> The config CometConf.COMET_SCHEMA_EVOLUTION_ENABLED is valid for Parquet files as well so removing it is not correct imo. I have added back `COMET_SCHEMA_EVOLUTION_ENABLED` > ScanRule is called at planning time while getColumnReader is called during execution. Why is the config not set correctly? This only happens when Comet Scan rule is not applied but Iceberg ParquetReaderType is set to Comet. Customer got this problem because in the previous version, `COMET_ENABLED` is off by default but Iceberg defaults ParquetReaderType to Comet. This problem should be gone in the latest Iceberg because now we take consideration of `COMET_ENABLED` when setting ParquetReaderType to Comet. However, user may still hit this problem when `COMET_ENABLED` sets to true but `COMET_NATIVE_SCAN_ENABLED` sets to false. > What types of schema evolution does iceberg require(i.e. support) for complex types? https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/types/TypeUtil.java#L402 Iceberg supports int to long, float to double and decimal precision promotion. If fields in complex type are int, float or decimal, schema evolution should be supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
