zhuqi-lucas opened a new issue, #23533: URL: https://github.com/apache/datafusion/issues/23533
### Describe the bug `DynamicFilterPhysicalExpr::current()` is called per batch on the RowFilter path (via `PhysicalExpr::evaluate`). It internally calls `remap_children`, which does a `transform_up` tree walk over the current expression. For dynamic filters carrying a large `InListExpr` — e.g. `HashJoinExec` pushing down `col IN build_keys` with ~150+ values — the walk is dominated by `InListExpr::with_new_children` cloning the whole list on every call. `inner.generation` only bumps on `update()` (once per HashJoin build, or once per TopK threshold refresh). Between updates every `current()` call recomputes an identical remapped tree. ### To Reproduce TPCH SF1, `pushdown_filters=true`, samply the Q17 hot path — `InListExpr::with_new_children` + its `drop_glue` sit at the top of the flamegraph. See #23420 for the wider TPCH regression context. ### Expected behavior `current()` should recompute the remap only when `inner.generation` changes — the walk output is deterministic within a generation. ### Impact TPCH SF1, 10 iterations: | Query | HEAD `pushdown=true` | Baseline `pushdown=false` | |---|---|---| | Q17 | 128 ms | 55 ms (2.4× gap) | | Total (22) | 743 ms | 531 ms | Q17 pays a **~73 ms per-batch remap tax** with `pushdown_filters=true` compared to the `pushdown=false` baseline; caching the remapped expression by generation is expected to close this gap. ### Additional context Part of #20324 (parquet filter-pushdown regression EPIC). PR #23532 fixes this specific overhead; the remaining TPCH gap (still ~110 ms after this fix) belongs to a separate two-phase-read regression that the adaptive-filter work in #22384 / #22237 targets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
