mbutrovich commented on code in PR #5107:
URL: https://github.com/apache/datafusion-comet/pull/5107#discussion_r3685107193
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -220,8 +225,30 @@ fn get_options(
allow_timestamp_ltz_to_ntz: bool,
object_store_url: &ObjectStoreUrl,
encryption_enabled: bool,
+ session_parquet_options: &ParquetOptions,
) -> (TableParquetOptions, SparkParquetOptions) {
let mut table_parquet_options = TableParquetOptions::new();
+
+ // Reader options that reach `ParquetSource`'s actual per-file execution
path
+ // (`ParquetMorselizer`, built from `table_parquet_options.global`) rather
than only
+ // `ParquetFormat::infer_schema`'s schema-inference path, which Comet
never calls (Comet
+ // always supplies its own schema, translated from Spark's catalog).
Seeded from the
+ // session so `spark.comet.datafusion.execution.parquet.*` (behind
+ // `spark.comet.exec.respectDataFusionConfigs`) and `spark.comet.parquet.
+ // rowFilterPushdown.enabled` actually take effect on the native scan
(#4990); a fresh
+ // `TableParquetOptions::new()` ignored the session entirely.
+ table_parquet_options.global.pushdown_filters =
session_parquet_options.pushdown_filters;
+ table_parquet_options.global.reorder_filters =
session_parquet_options.reorder_filters;
+ table_parquet_options.global.force_filter_selections =
+ session_parquet_options.force_filter_selections;
+ table_parquet_options.global.bloom_filter_on_read =
+ session_parquet_options.bloom_filter_on_read;
+ table_parquet_options.global.max_predicate_cache_size =
+ session_parquet_options.max_predicate_cache_size;
+ table_parquet_options.global.enable_page_index =
session_parquet_options.enable_page_index;
Review Comment:
Should be addressed in
https://github.com/apache/datafusion-comet/pull/5107/commits/08017243f08f488f62b82402a3763388046956c0,
thanks @andygrove!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
