westonpace commented on code in PR #14194:
URL: https://github.com/apache/datafusion/pull/14194#discussion_r1975891476
##########
datafusion/substrait/src/logical_plan/producer.rs:
##########
@@ -559,12 +559,31 @@ pub fn from_table_scan(
let table_schema = scan.source.schema().to_dfschema_ref()?;
let base_schema = to_substrait_named_struct(&table_schema)?;
+ let best_effort_filter_option = if !scan.filters.is_empty() {
+ let table_schema_qualified = Arc::new(
+ DFSchema::try_from_qualified_schema(
+ scan.table_name.clone(),
+ &(scan.source.schema()),
+ )
+ .unwrap(),
+ );
+ let mut combined_expr = scan.filters[0].clone();
+ for i in 1..scan.filters.len() {
+ combined_expr = combined_expr.and(scan.filters[i].clone());
+ }
+ let best_effort_filter_expr =
+ producer.handle_expr(&combined_expr, &table_schema_qualified)?;
+ Some(Box::new(best_effort_filter_expr))
+ } else {
+ None
+ };
+
Ok(Box::new(Rel {
rel_type: Some(RelType::Read(Box::new(ReadRel {
common: None,
base_schema: Some(base_schema),
filter: None,
- best_effort_filter: None,
+ best_effort_filter: best_effort_filter_option,
Review Comment:
> During logical plan optimization, the predicate from Filter will get
pushed down to TableScan
I'm probably out of my depth here. What is `Logical plan optimization`?
From your post it appears to be something that takes one logical plan as input
and returns a more optimized logical plan as input? It is very possible that
something like this exists (many query engines have this), but if it does then
I haven't worked with it and so I don't know exactly what it is capable of.
In the second logical plan you have:
```
TableScan:
table_name: t1
filters: [(c1>0, Inexact)]
```
What does this actually look like in the logical plan. If I look at
`TableScan::filter` then I see it is a `Vec<Expr>`. How is `(c1>0, Inexact)`
represented? Is there an `Expr` node that represents "inexact"?
Here is my understanding. To start with there is the logical plan:
```
Projection:
expr: c1
Filter:
predicate: (c1 > 0)
TableScan:
table_name: t1
filters: None
...
```
In a pure-logical planner I would expect the "push all filters down as far
as you can" output to be:
```
Projection:
expr: c1
TableScan:
table_name: t1
filters: [c1 > 0]
```
Then, during physical planning, assuming the table provider reports Inexact,
I would expect the output physical plan to be:
```
ProjectExec
FilterExec
expr: c1 > 0
FooScanExec
filters: [c1 > 0]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
