mbutrovich commented on code in PR #5107:
URL: https://github.com/apache/datafusion-comet/pull/5107#discussion_r3685109055
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -76,6 +76,10 @@ pub(crate) fn init_datasource_exec(
use_field_id: bool,
ignore_missing_field_id: bool,
) -> Result<Arc<DataSourceExec>, ExecutionError> {
+ // Computed once and reused below for `try_pushdown_filters`, since
`SessionContext::state()`
+ // clones the whole `SessionState` (function registries, optimizer rules,
etc.), not just the
+ // config.
+ let state = session_ctx.state();
Review Comment:
Should be addressed in
https://github.com/apache/datafusion-comet/pull/5107/commits/08017243f08f488f62b82402a3763388046956c0,
thanks @andygrove!
##########
native/core/src/execution/jni_api.rs:
##########
@@ -590,11 +590,22 @@ fn prepare_datafusion_session_context(
// Translate the Comet-namespaced row-level pushdown flag into the
equivalent
// DataFusion session options. `pushdown_filters` enables the parquet
reader's
// RowFilter evaluation during decode (late materialization);
`reorder_filters`
- // is only meaningful when pushdown_filters is on, so they move together.
+ // is only meaningful when pushdown_filters is on, so they move together.
Skip a
+ // key the `spark.comet.datafusion.*` testing escape hatch above already
set
+ // explicitly, so an explicit override isn't silently forced back to
`true`.
if spark_config.get_bool(COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED) {
- session_config = session_config
- .set_str("datafusion.execution.parquet.pushdown_filters", "true")
- .set_str("datafusion.execution.parquet.reorder_filters", "true");
+ const PUSHDOWN_FILTERS_KEY: &str =
+ "spark.comet.datafusion.execution.parquet.pushdown_filters";
+ const REORDER_FILTERS_KEY: &str =
+ "spark.comet.datafusion.execution.parquet.reorder_filters";
+ if !spark_config.contains_key(PUSHDOWN_FILTERS_KEY) {
Review Comment:
Should be addressed in
https://github.com/apache/datafusion-comet/pull/5107/commits/08017243f08f488f62b82402a3763388046956c0,
thanks @andygrove!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
