oglego opened a new pull request, #800:
URL: https://github.com/apache/hudi-rs/pull/800
## Description
This PR is for the TODO in `file_pruner.rs`:
```rust
// Multi-value operators not yet supported for file-level pruning
// TODO: support IN/NOT IN by checking if all values are outside the min/max
range
```
The `can_prune_by_filter` skipped IN/NOT IN filters (a TODO returned "don't
prune" for all multi-value operators).
This PR adds file-level pruning from column min/max stats:
- `col IN (v1, v2, ...)`: prune when every value is outside [min, max]
(equivalent to pruning on each `col = vk`; bounds are inclusive).
- `col NOT IN (v1, v2, ...)`: prune only when min == max and that value is
in the
list (i.e. every row is an excluded value). Implemented by delegating to
the
existing `!=` rule per value.
## How are the changes test-covered
- [ ] N/A
- Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
New unit tests in `file_pruner.rs` cover the below:
- values fully outside the range
- on both sides of the range, and inside it
- inclusive min/max boundaries
- single-value IN
- string columns
- NOT IN with min == max (excluded and not excluded) and with min != max
where both bounds are excluded
- missing min/max (both, and one bound at a time)
- an uncastable value in the list
- combined IN + NOT IN filters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
