mazne opened a new issue, #27513:
URL: https://github.com/apache/datafusion/issues/27513
**Versions:** DataFusion 55.2.0, arrow / parquet 59.3.2 (macOS, Apple
Silicon).
**What happens**
DataFusion 55 re-evaluates a TopK dynamic filter at each row-group boundary
of an open Parquet file and skips row groups the current threshold rules out
(`row_groups_pruned_dynamic_filter`). This works when the table schema matches
the file. It never fires when the file stores the sort column as `REQUIRED` and
the table schema declares the same column nullable. That is common when a table
is given an explicit, all-nullable schema so that files written with and
without a column can be read as one table.
The rows are still filtered by the threshold, so results are correct. The
loss is that every row group of the file is decoded.
**Repro** (dependencies: `datafusion`, `arrow`, `parquet`, `tokio`,
`tempfile`)
```rust
use std::sync::Arc;
use arrow::array::{Int64Array, RecordBatch};
use arrow::datatypes::{DataType, Field, Schema};
use datafusion::physical_plan::{collect, display::DisplayableExecutionPlan};
use datafusion::prelude::*;
use parquet::arrow::ArrowWriter;
use parquet::file::properties::WriterProperties;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
// One file, 1M rows in 16 row groups; `v` rises with the row, plus
noise.
let dir = tempfile::tempdir()?;
let values: Vec<i64> = (0..1_000_000_i64).map(|i| i + (i * 7_919) %
50_000).collect();
let required = Arc::new(Schema::new(vec![Field::new("v",
DataType::Int64, false)]));
let batch = RecordBatch::try_new(required.clone(),
vec![Arc::new(Int64Array::from(values))])?;
let props =
WriterProperties::builder().set_max_row_group_row_count(Some(65_536)).build();
let mut writer =
ArrowWriter::try_new(std::fs::File::create(dir.path().join("a.parquet"))?,
required, Some(props))?;
writer.write(&batch)?;
writer.close()?;
let nullable = Schema::new(vec![Field::new("v", DataType::Int64, true)]);
for (label, options) in [
("table schema as in the file (required)",
ParquetReadOptions::default()),
("table schema nullable",
ParquetReadOptions::default().schema(&nullable)),
] {
// One partition keeps the file in one range, so all 16 groups are
in one scan.
let ctx =
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(1));
ctx.register_parquet("t", dir.path().to_str().unwrap(),
options).await?;
let plan = ctx
.table("t")
.await?
.sort(vec![col("v").sort(true, true)])?
.limit(0, Some(100))?
.create_physical_plan()
.await?;
collect(plan.clone(), ctx.task_ctx()).await?;
let text =
DisplayableExecutionPlan::with_metrics(plan.as_ref()).indent(true).to_string();
let scan = text.lines().find(|l|
l.contains("DataSourceExec")).unwrap_or("");
let keep: Vec<&str> = scan
.split(", ")
.filter(|m| m.contains("row_groups_pruned_dynamic_filter") ||
m.contains("bytes_scanned"))
.collect();
println!("{label}: {keep:?}");
}
Ok(())
}
```
**Expected:** the same pruning for both table schemas. The file holds no
nulls, and a nullable table schema should not change which row groups can hold
the top 100.
**Actual:**
```
table schema as in the file (required): ["bytes_scanned=99.30 K",
"row_groups_pruned_dynamic_filter=15"]
table schema nullable: ["bytes_scanned=1.52 M",
"row_groups_pruned_dynamic_filter=0"]
```
This is the same with `datafusion.execution.parquet.pushdown_filters` on or
off. With more target partitions the file is split into one-group ranges and
neither case prunes, which is a separate matter.
`num_predicate_creation_errors` and `predicate_evaluation_errors` are 0 in both
cases.
A variant with the column `OPTIONAL` in the file and the table nullable
prunes like the matching case, so the trigger is the mismatch: the file's
column is non-nullable and the table's is nullable.
**Where I suspect the cause**
The per-file predicate rewrite in the Parquet opener
(`datafusion-datasource-parquet/src/opener/mod.rs`, where the expression
adapter and `PhysicalExprSimplifier` rewrite `prepared.predicate` against the
physical file schema). The rewritten predicate differs only when the file's
column is non-nullable: the TopK filter `v IS NULL OR v < t` can simplify `v IS
NULL` to `false`.
The runtime row-group pruner is built only when
`DynamicFilterTracking::classify(predicate)` returns `Watching`, and its
re-checks rely on that subscription. My guess is that the rewritten form no
longer reaches it as a watched dynamic filter. Either no pruner is built, or it
never sees the threshold change. I have not confirmed which.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]