lets-order-some-fries commented on issue #65606:
URL: https://github.com/apache/doris/issues/65606#issuecomment-5444017878
Root-caused this against current master. The mechanism, plus one narrowing
that I think matters for choosing the fix.
### Why the partition is pruned
The row lands in `p_2000`, not `p_2001`: its partition key is
`date_trunc('2001-01-07','month')` = `2001-01-01`, which falls in `p_2000`'s
range `['2000-01-01','2001-01-05')`. Pruning then evaluates `TIME_STAMP >
'2001-01-06'` against those same bounds *as if they were raw-column values*.
`p_2000`'s upper bound is below the predicate's lower bound, so the partition
is pruned — and the row goes with it.
`PruneOlapScanPartition.getPartitionSlots` builds the slots to prune on from
`partitionInfo.getPartitionColumns()`, matching them to scan output by name
(`PruneOlapScanPartition.java:191-204`). `getPartitionExprs()` is never
consulted on the read path — its only use under `nereids/` is insert-side row
routing in `BindSink`. So the transform between the two domains isn't
represented anywhere the pruner can see.
### Aligned partitions are safe; only misaligned ones leak
Auto-created partitions cannot hit this. `PartitionExprUtil.getRangeEnd`
builds each as `[begin, begin.plusMonths(interval))` with `begin` already the
truncated value, so an auto-created partition's raw-value contents exactly
equal its bound range, and raw-column pruning is sound.
The repro's `p_2000` = `['2000-01-01','2001-01-05')` is manually created and
*not* month-aligned. Any raw date whose month-truncation is still `<
2001-01-05` lands in it — everything up to `2001-01-31` — so the partition
holds values well beyond its stated upper bound. That overhang is exactly what
pruning discards.
So the bug requires a partition whose bounds are not aligned to the
partition expression's granularity, which today only a manual `PARTITION ...
VALUES [...)` clause can produce.
### A guard for this hazard already exists, on a different path
`RuntimeFilterPruneClassifier` already refuses RF-based partition pruning
for this exact situation — `hasUnsupportedAutomaticPartitionExpression()`
(`RuntimeFilterPruneClassifier.java:260-268`), called at `:151` with the reason
*"automatic partition expression boundary is not modeled"*. The main pruner has
no equivalent.
Both main strategies funnel through `PartitionPruner.pruneWithResult(...)`
at `PruneOlapScanPartition.java:180-182` — the per-partition evaluator and the
`SortedPartitionRanges` binary-search path — so a guard placed there would
cover both. The binary search matters here because it only understands "bare
column op literal" shapes, per the note at
`InferPredicateFromMonotonicFunction.java:58`.
### The part I don't think an outside contributor should decide alone
1. **Mirror the RF guard** — skip pruning whenever the table is
auto-partitioned by a function expression. Correct, but it gives up partition
pruning for every `AUTO PARTITION BY RANGE(date_trunc(...))` table, including
the aligned majority. Real performance regression.
2. **Skip pruning only for partitions whose bounds aren't
granularity-aligned.** Keeps the common case fast; needs an alignment check
against the partition expression.
3. **Reject misaligned manual partitions at DDL time**, so the state cannot
arise. Cleanest semantics, but breaking for existing tables.
I'd lean to (2) as the best correctness/performance trade, but it's a
semantics call that belongs with the maintainers. Happy to implement whichever
you prefer, with an FE unit test asserting the pruner's surviving-partition set
for both the aligned and misaligned cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
