alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2660854312
Thanks again @UBarney and @adriangb and @findepi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
alamb merged PR #14567:
URL: https://github.com/apache/datafusion/pull/14567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusi
alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2659119191
I'll plan to merge this PR later today or tomorrow unless anyone would like
additional time to review
--
This is an automated message from the Apache Git Service.
To respond to the m
alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2659118339
> However, all values match the predicate so this should not be pruned.
I agree -- the transformation as specifically proposed by @findepi would
incorrectly filter out some ran
UBarney commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2658014801
@alamb Here's an example where @findepi's transformation would prune a
container, but this won't.
Given data
| col |
|-|
| a |
| b |
| z |
UBarney commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1954594478
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -4061,6 +4132,106 @@ mod tests {
prune_with_expr(expr, &schema, &statistics, expected_ret);
}
alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2656655495
> `t.parquet` in is [zip
file](https://github.com/user-attachments/files/18784196/t.zip) (github doesn't
allow upload parquet file 😓)
BTW you can create such files using datafu
alamb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1954515692
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,76 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'const_prefi
alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2656570666
> @alamb Thanks for such a detailed example and explanation. I think this
transformation is correct in this case.
>
> according to this
[doc](https://docs.rs/datafusion/latest/d
UBarney commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2656543100
@alamb
This transformation is correct in this case
according to this
[doc](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.h
alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2655007520
This is on my list of PRs to review tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
UBarney commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1952755081
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,76 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'const_pre
UBarney commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1952755081
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,76 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'const_pre
findepi commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1952641084
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,76 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'const_pre
UBarney commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1952625400
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,66 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`, w
adriangb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1951517180
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,66 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`,
findepi commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1951514644
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,66 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`, w
findepi commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1951508960
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1711,66 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`, w
adriangb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1951264573
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`,
findepi commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1950837919
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`, w
adriangb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1950314013
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`,
adriangb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1950314013
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`,
UBarney commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1950259492
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`, w
UBarney commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1950259274
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1590,6 +1590,13 @@ fn build_statistics_expr(
)),
))
}
+Opera
UBarney commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2649842916
> I assume this should also work with no wildcards `col not like 'foo'`?
Yes. add some test to demonstrate it
There's also an optimization to rewrite `col not like 'foo'` as
adriangb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1949194705
##
datafusion/physical-optimizer/src/pruning.rs:
##
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`,
alamb commented on PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#issuecomment-2648116457
FYI @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
UBarney opened a new pull request, #14567:
URL: https://github.com/apache/datafusion/pull/14567
## Which issue does this PR close?
- Closes #14053 .
## Rationale for this change
## What changes are included in this PR?
If both col_min and col_max match
28 matches
Mail list logo
