schenksj opened a new issue, #27543:
URL: https://github.com/apache/datafusion/issues/27543
### Describe the bug
A physical plan that has already been executed can still be serialized with
`datafusion-proto`. When it is, every `DynamicFilterPhysicalExpr` in it (the
TopK filter held by `SortExec`, and the copy pushed into the `DataSourceExec`
predicate) is encoded with its **current** inner expression, `generation` and
`is_complete`, that is, the final threshold of the run that just finished.
`DynamicFilterPhysicalExpr::try_to_proto` reads `self.inner` as is.
Decoding that plan and executing it again applies the stale threshold from
the start. Because the TopK threshold is a strict bound that excludes the rows
that set it, the re-run returns too few rows. No error is raised.
For `SELECT * FROM t ORDER BY id LIMIT 1` over ids 1..=200 the decoded plan
is
```
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[false], filter=[id@0 < 1]
DataSourceExec: ..., predicate=DynamicFilter [ id@0 < 1 ], ...,
pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 < 1, ...
```
and returns 0 rows instead of 1. `ORDER BY id DESC LIMIT 3` decodes with
`filter=[id@0 IS NULL OR id@0 > 198]` and returns 2 rows instead of 3.
We hit this when distributing plans: a coordinator plans and runs a query,
then ships the same `Arc<dyn ExecutionPlan>` to workers.
### To Reproduce
`Cargo.toml`:
```toml
[dependencies]
datafusion = "=55.3.0"
datafusion-proto = "=55.3.0"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
```
`src/main.rs`:
```rust
//! Minimal repro: a physical plan serialized with datafusion-proto *after*
it has executed
//! carries the TopK DynamicFilterPhysicalExpr at its final runtime value.
use std::sync::Arc;
use datafusion::arrow::array::{Int64Array, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::parquet::file::properties::WriterProperties;
use datafusion::physical_plan::{collect, displayable, ExecutionPlan};
use datafusion::prelude::*;
use datafusion_proto::bytes::{physical_plan_from_bytes,
physical_plan_to_bytes};
/// Writes `files` Parquet files holding ids 1..=200 between them, 25 rows
per row group.
fn write_table(dir: &std::path::Path, files: usize) ->
datafusion::error::Result<()> {
let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64,
false)]));
let per_file = 200 / files as i64;
for f in 0..files as i64 {
let file =
std::fs::File::create(dir.join(format!("part-{f}.parquet")))?;
let props =
WriterProperties::builder().set_max_row_group_size(25).build();
let mut w = ArrowWriter::try_new(file, schema.clone(), Some(props))?;
let ids: Vec<i64> = (f * per_file + 1..=(f + 1) *
per_file).collect();
for chunk in ids.chunks(25) {
let batch = RecordBatch::try_new(schema.clone(),
vec![Arc::new(Int64Array::from(chunk.to_vec()))])?;
w.write(&batch)?;
}
w.close()?;
}
Ok(())
}
async fn new_ctx(dir: &std::path::Path) ->
datafusion::error::Result<SessionContext> {
let ctx = SessionContext::new();
ctx.register_parquet("t", dir.to_str().unwrap(),
ParquetReadOptions::default()).await?;
Ok(ctx)
}
fn show(label: &str, plan: &Arc<dyn ExecutionPlan>) {
println!("--- {label} ---\n{}", displayable(plan.as_ref()).indent(true));
}
fn rows(batches: &[RecordBatch]) -> usize {
batches.iter().map(|b| b.num_rows()).sum()
}
async fn run_case(sql: &str, files: usize) -> datafusion::error::Result<()> {
println!("\n================ {sql} ({files} file(s))
================");
let dir =
std::env::temp_dir().join(format!("df-dynfilter-repro-{files}"));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir)?;
write_table(&dir, files)?;
// Control: round-trip the plan BEFORE executing it.
let ctx = new_ctx(&dir).await?;
let plan = ctx.sql(sql).await?.create_physical_plan().await?;
let bytes = physical_plan_to_bytes(plan.clone())?;
let ctx2 = new_ctx(&dir).await?;
let decoded = physical_plan_from_bytes(&bytes, &ctx2.task_ctx())?;
let n = rows(&collect(decoded, ctx2.task_ctx()).await?);
println!("control (encode before execute): decoded plan returned {n}
row(s)");
// Bug case: execute the plan, then encode the same (executed) plan.
let ctx = new_ctx(&dir).await?;
let plan = ctx.sql(sql).await?.create_physical_plan().await?;
show("plan before execution", &plan);
let n = rows(&collect(plan.clone(), ctx.task_ctx()).await?);
println!("original plan returned {n} row(s)");
show("same plan after execution", &plan);
let bytes = physical_plan_to_bytes(plan.clone())?;
let ctx2 = new_ctx(&dir).await?;
let decoded = physical_plan_from_bytes(&bytes, &ctx2.task_ctx())?;
show("decoded plan (encoded after execution)", &decoded);
let n2 = rows(&collect(decoded, ctx2.task_ctx()).await?);
println!("decoded plan returned {n2} row(s) [expected {n}] => {}",
if n2 == n { "OK" } else { "WRONG RESULT" });
Ok(())
}
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
run_case("SELECT * FROM t ORDER BY id LIMIT 1", 1).await?;
run_case("SELECT * FROM t ORDER BY id LIMIT 1", 4).await?;
run_case("SELECT * FROM t ORDER BY id DESC LIMIT 3", 4).await?;
Ok(())
}
```
Output (55.3.0; 55.2.0 is identical):
```
================ SELECT * FROM t ORDER BY id LIMIT 1 (1 file(s))
================
control (encode before execute): decoded plan returned 1 row(s)
--- plan before execution ---
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[false]
DataSourceExec: file_groups={1 group:
[[tmp/df-dynfilter-repro-1/part-0.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ empty ],
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible
original plan returned 1 row(s)
--- same plan after execution ---
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[false], filter=[id@0 < 1]
DataSourceExec: file_groups={1 group:
[[tmp/df-dynfilter-repro-1/part-0.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ],
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible,
pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 < 1,
required_guarantees=[]
--- decoded plan (encoded after execution) ---
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[false], filter=[id@0 < 1]
DataSourceExec: file_groups={1 group:
[[tmp/df-dynfilter-repro-1/part-0.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ],
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2
AND id_min@0 < 1, required_guarantees=[]
decoded plan returned 0 row(s) [expected 1] => WRONG RESULT
================ SELECT * FROM t ORDER BY id LIMIT 1 (4 file(s))
================
control (encode before execute): decoded plan returned 1 row(s)
--- plan before execution ---
SortPreservingMergeExec: [id@0 ASC NULLS LAST], fetch=1
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[true]
DataSourceExec: file_groups={4 groups:
[[tmp/df-dynfilter-repro-4/part-0.parquet],
[tmp/df-dynfilter-repro-4/part-1.parquet],
[tmp/df-dynfilter-repro-4/part-2.parquet],
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ empty ],
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible
original plan returned 1 row(s)
--- same plan after execution ---
SortPreservingMergeExec: [id@0 ASC NULLS LAST], fetch=1
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[true], filter=[id@0 < 1]
DataSourceExec: file_groups={4 groups:
[[tmp/df-dynfilter-repro-4/part-0.parquet],
[tmp/df-dynfilter-repro-4/part-1.parquet],
[tmp/df-dynfilter-repro-4/part-2.parquet],
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ],
sort_order_for_reorder=[id@0 ASC NULLS LAST], dynamic_rg_pruning=eligible,
pruning_predicate=id_null_count@1 != row_count@2 AND id_min@0 < 1,
required_guarantees=[]
--- decoded plan (encoded after execution) ---
SortPreservingMergeExec: [id@0 ASC NULLS LAST], fetch=1
SortExec: TopK(fetch=1), expr=[id@0 ASC NULLS LAST],
preserve_partitioning=[true], filter=[id@0 < 1]
DataSourceExec: file_groups={4 groups:
[[tmp/df-dynfilter-repro-4/part-0.parquet],
[tmp/df-dynfilter-repro-4/part-1.parquet],
[tmp/df-dynfilter-repro-4/part-2.parquet],
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ id@0 < 1 ],
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2
AND id_min@0 < 1, required_guarantees=[]
decoded plan returned 0 row(s) [expected 1] => WRONG RESULT
================ SELECT * FROM t ORDER BY id DESC LIMIT 3 (4 file(s))
================
control (encode before execute): decoded plan returned 3 row(s)
--- plan before execution ---
SortPreservingMergeExec: [id@0 DESC], fetch=3
SortExec: TopK(fetch=3), expr=[id@0 DESC], preserve_partitioning=[true]
DataSourceExec: file_groups={4 groups:
[[tmp/df-dynfilter-repro-4/part-0.parquet],
[tmp/df-dynfilter-repro-4/part-1.parquet],
[tmp/df-dynfilter-repro-4/part-2.parquet],
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ empty ],
sort_order_for_reorder=[id@0 DESC], reverse_row_groups=true,
dynamic_rg_pruning=eligible
original plan returned 3 row(s)
--- same plan after execution ---
SortPreservingMergeExec: [id@0 DESC], fetch=3
SortExec: TopK(fetch=3), expr=[id@0 DESC], preserve_partitioning=[true],
filter=[id@0 IS NULL OR id@0 > 198]
DataSourceExec: file_groups={4 groups:
[[tmp/df-dynfilter-repro-4/part-0.parquet],
[tmp/df-dynfilter-repro-4/part-1.parquet],
[tmp/df-dynfilter-repro-4/part-2.parquet],
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 > 198 ],
sort_order_for_reorder=[id@0 DESC], reverse_row_groups=true,
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR
id_null_count@0 != row_count@2 AND id_max@1 > 198, required_guarantees=[]
--- decoded plan (encoded after execution) ---
SortPreservingMergeExec: [id@0 DESC], fetch=3
SortExec: TopK(fetch=3), expr=[id@0 DESC], preserve_partitioning=[true],
filter=[id@0 IS NULL OR id@0 > 198]
DataSourceExec: file_groups={4 groups:
[[tmp/df-dynfilter-repro-4/part-0.parquet],
[tmp/df-dynfilter-repro-4/part-1.parquet],
[tmp/df-dynfilter-repro-4/part-2.parquet],
[tmp/df-dynfilter-repro-4/part-3.parquet]]}, projection=[id],
file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 > 198 ],
dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR
id_null_count@0 != row_count@2 AND id_max@1 > 198, required_guarantees=[]
decoded plan returned 2 row(s) [expected 3] => WRONG RESULT
```
The control (encode the plan before executing it) returns the right row
count in every case.
### Expected behavior
Serializing a plan should give the same result whether or not that plan
object has run in this process. The decoded plan should return 1 row for `ORDER
BY id LIMIT 1` (and 3 for `ORDER BY id DESC LIMIT 3`), exactly as when the plan
is encoded before it runs.
### Additional context
- `DynamicFilterPhysicalExpr::try_to_proto`
(`datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs`) encodes
`self.inner.read().clone()`, so `inner_expr`, `generation` and `is_complete`
are the runtime values.
- Calling `datafusion::physical_plan::execution_plan::reset_plan_states`
before encoding works around it for the filter a node produces. As documented,
it does not reset a dynamic filter that planning pushed into a child (for
example a hash join's filter pushed into a Parquet scan), so it is not a full
fix.
- Possible fixes:
1. Serialize dynamic filters in their initial state: the placeholder
expression (`lit(true)`), generation 0 and `is_complete = false`. Runtime state
is per execution and does not belong in a plan. This needs the initial
expression to be kept on the `DynamicFilterPhysicalExpr`, or recoverable from
it.
2. Alternatively, expose the initial state, e.g. a public
`DynamicFilterPhysicalExpr::reset()` or `initial_expr()`, so a
`PhysicalProtoConverterExtension` or a `reset_plan_states`-style pass can
restore it, including for copies pushed into children.
- Possibly related: #19905 (dynamic filters are not reset by
`reset_plan_states`), #21650 (shared mutable state on `ExecutionPlan`), #27130
(reusing a `limit=1` plan returns an empty result), #25154 (dynamic filter
identity across serialization). This may belong under #26672.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]