mazne opened a new issue, #27513:
URL: https://github.com/apache/datafusion/issues/27513

   **Versions:** DataFusion 55.2.0, arrow / parquet 59.3.2 (macOS, Apple 
Silicon).
   
   **What happens**
   
   DataFusion 55 re-evaluates a TopK dynamic filter at each row-group boundary 
of an open Parquet file and skips row groups the current threshold rules out 
(`row_groups_pruned_dynamic_filter`). This works when the table schema matches 
the file. It never fires when the file stores the sort column as `REQUIRED` and 
the table schema declares the same column nullable. That is common when a table 
is given an explicit, all-nullable schema so that files written with and 
without a column can be read as one table.
   
   The rows are still filtered by the threshold, so results are correct. The 
loss is that every row group of the file is decoded.
   
   **Repro** (dependencies: `datafusion`, `arrow`, `parquet`, `tokio`, 
`tempfile`)
   
   ```rust
   use std::sync::Arc;
   
   use arrow::array::{Int64Array, RecordBatch};
   use arrow::datatypes::{DataType, Field, Schema};
   use datafusion::physical_plan::{collect, display::DisplayableExecutionPlan};
   use datafusion::prelude::*;
   use parquet::arrow::ArrowWriter;
   use parquet::file::properties::WriterProperties;
   
   #[tokio::main]
   async fn main() -> datafusion::error::Result<()> {
       // One file, 1M rows in 16 row groups; `v` rises with the row, plus 
noise.
       let dir = tempfile::tempdir()?;
       let values: Vec<i64> = (0..1_000_000_i64).map(|i| i + (i * 7_919) % 
50_000).collect();
       let required = Arc::new(Schema::new(vec![Field::new("v", 
DataType::Int64, false)]));
       let batch = RecordBatch::try_new(required.clone(), 
vec![Arc::new(Int64Array::from(values))])?;
       let props = 
WriterProperties::builder().set_max_row_group_row_count(Some(65_536)).build();
       let mut writer = 
ArrowWriter::try_new(std::fs::File::create(dir.path().join("a.parquet"))?, 
required, Some(props))?;
       writer.write(&batch)?;
       writer.close()?;
   
       let nullable = Schema::new(vec![Field::new("v", DataType::Int64, true)]);
       for (label, options) in [
           ("table schema as in the file (required)", 
ParquetReadOptions::default()),
           ("table schema nullable", 
ParquetReadOptions::default().schema(&nullable)),
       ] {
           // One partition keeps the file in one range, so all 16 groups are 
in one scan.
           let ctx = 
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(1));
           ctx.register_parquet("t", dir.path().to_str().unwrap(), 
options).await?;
           let plan = ctx
               .table("t")
               .await?
               .sort(vec![col("v").sort(true, true)])?
               .limit(0, Some(100))?
               .create_physical_plan()
               .await?;
           collect(plan.clone(), ctx.task_ctx()).await?;
           let text = 
DisplayableExecutionPlan::with_metrics(plan.as_ref()).indent(true).to_string();
           let scan = text.lines().find(|l| 
l.contains("DataSourceExec")).unwrap_or("");
           let keep: Vec<&str> = scan
               .split(", ")
               .filter(|m| m.contains("row_groups_pruned_dynamic_filter") || 
m.contains("bytes_scanned"))
               .collect();
           println!("{label}: {keep:?}");
       }
       Ok(())
   }
   ```
   
   **Expected:** the same pruning for both table schemas. The file holds no 
nulls, and a nullable table schema should not change which row groups can hold 
the top 100.
   
   **Actual:**
   
   ```
   table schema as in the file (required): ["bytes_scanned=99.30 K", 
"row_groups_pruned_dynamic_filter=15"]
   table schema nullable: ["bytes_scanned=1.52 M", 
"row_groups_pruned_dynamic_filter=0"]
   ```
   
   This is the same with `datafusion.execution.parquet.pushdown_filters` on or 
off. With more target partitions the file is split into one-group ranges and 
neither case prunes, which is a separate matter. 
`num_predicate_creation_errors` and `predicate_evaluation_errors` are 0 in both 
cases.
   
   A variant with the column `OPTIONAL` in the file and the table nullable 
prunes like the matching case, so the trigger is the mismatch: the file's 
column is non-nullable and the table's is nullable.
   
   **Where I suspect the cause**
   
   The per-file predicate rewrite in the Parquet opener 
(`datafusion-datasource-parquet/src/opener/mod.rs`, where the expression 
adapter and `PhysicalExprSimplifier` rewrite `prepared.predicate` against the 
physical file schema). The rewritten predicate differs only when the file's 
column is non-nullable: the TopK filter `v IS NULL OR v < t` can simplify `v IS 
NULL` to `false`.
   
   The runtime row-group pruner is built only when 
`DynamicFilterTracking::classify(predicate)` returns `Watching`, and its 
re-checks rely on that subscription. My guess is that the rewritten form no 
longer reaches it as a watched dynamic filter. Either no pruner is built, or it 
never sees the threshold change. I have not confirmed which.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to