This is an automated email from the ASF dual-hosted git repository.
924060929 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3cfbf220312 [fix](nereids) fix limit + offset overflow when pushing
down TopN/Limit (#64633)
3cfbf220312 is described below
commit 3cfbf220312f601e87dcdedb3b8efbdb0882e4bd
Author: 924060929 <[email protected]>
AuthorDate: Thu Jul 2 11:30:04 2026 +0800
[fix](nereids) fix limit + offset overflow when pushing down TopN/Limit
(#64633)
When combining a `limit` and an `offset` into the number of rows a child
must keep — pushing a
`TopN`/`Limit` down, building a two-phase sort, splitting a limit into
local + global phases, etc. —
the raw `limit + offset` overflows the `long` range when both are close
to `BIGINT_MAX`
(e.g. `LIMIT 9223372036854775807 OFFSET 9223372036854775807`) and wraps
to a negative value.
A negative limit is an illegal plan. On the BE side it is reinterpreted
as a huge unsigned value
(`uint64_t limit = _offset + _limit` in the sorter / `_heap_size(limit +
offset)` in HeapSorter),
so a trivial query that should immediately return an empty set instead
runs until it hits the query
timeout, or triggers UBSan (signed integer overflow is undefined
behavior in C++).
### Minimal reproducer (no table required)
```sql
select count(*) as c from (
select id from (
select 1 as id union all select 2 as id union all select 3 as id
) t
order by id limit 9223372036854775807 offset 9223372036854775807
) s;
```
- Original planner, or Nereids with `PUSH_DOWN_TOP_N_THROUGH_UNION`
disabled: returns `0` immediately.
- Nereids with the rule enabled: times out (before this fix).
### Fix
Add `Utils.addOverflows(long, long)` to detect when `limit + offset`
exceeds the `long` range.
On overflow, the behavior depends on whether the code path can be
declined:
**Skip the optimization** (can decline — the unoptimized plan is
correct):
- `TopN`/`Limit` push-down through `Union` / `Join` / `Project-Join` /
`Window` and their distinct
variants skip the rewrite; the parent `TopN`/`Limit` still applies the
original `limit`/`offset`.
- The `topn_opt_limit_threshold` gates (`PushDownTopNThroughJoin` /
`PushDownTopNDistinctThroughJoin` /
`PushDownLimitDistinctThroughJoin` / `LimitAggToTopNAgg` /
`PushTopnToAgg`) treat overflow as
over-threshold and do not fire.
- The score / vector `TopN`-into-`OlapScan` push-downs and the
sort-limit-into-`OlapScan` translation
in `PhysicalPlanTranslator` are skipped.
- `CollectLimitAboveConsumer` skips recording the consumer row count on
overflow, leaving the CTE
producer unbounded (the consumer's own limit still applies).
**Report an error** (cannot decline — overflowing `limit + offset`
cannot be correctly represented):
- `LogicalTopNToPhysicalTopN`: the two-phase local sort needs `limit +
offset` rows, and even a
single-phase `GATHER_SORT` would pass the overflowing values to BE where
`HeapSorter` also computes
`limit + offset` (triggering UBSan). Since no valid physical plan can be
produced, report an error.
- `SplitLimit`: the two-phase local limit needs `limit + offset`; an
unsplit `ORIGIN` limit is silently
dropped during translation, and `Long.MAX_VALUE` is not treated as
"unlimited" by the legacy
`PlanNode` layer (`hasLimit()` returns `limit > -1`). Report an error.
- `MergeLimits.mergeOffset` / `MergeTopNs`: consecutive limits/TopNs
must be merged, and the merged
offset (`upperOffset + bottomOffset`) cannot be represented on overflow.
Report an error.
For non-overflowing inputs the behavior is unchanged.
**Defense-in-depth**: the `LogicalTopN` / `PhysicalTopN` /
`LogicalLimit` / `PhysicalLimit`
constructors now assert that `limit` and `offset` are non-negative. Any
future path that reintroduces
an overflowing `limit + offset` fails fast during planning instead of
silently hanging in BE.
### Tests
- `UtilsTest#testAddOverflows` covers the overflow detection boundary.
- Regression cases in `push_down_top_n_through_union` assert that the
reproducer (both TopN and Limit
paths with BIGINT_MAX limit/offset) reports an error instead of timing
out.
---
.../glue/translator/PhysicalPlanTranslator.java | 8 ++++++-
.../nereids/processor/post/PushTopnToAgg.java | 6 +++++
.../implementation/LogicalTopNToPhysicalTopN.java | 15 +++++++++---
.../rules/rewrite/CollectLimitAboveConsumer.java | 13 +++++++++--
.../nereids/rules/rewrite/LimitAggToTopNAgg.java | 5 ++++
.../doris/nereids/rules/rewrite/MergeLimits.java | 11 +++++++++
.../doris/nereids/rules/rewrite/MergeTopNs.java | 6 +++++
.../rewrite/PushDownLimitDistinctThroughJoin.java | 8 +++++++
.../rewrite/PushDownLimitDistinctThroughUnion.java | 7 ++++++
.../rewrite/PushDownScoreTopNIntoOlapScan.java | 7 ++++++
.../rewrite/PushDownTopNDistinctThroughJoin.java | 18 +++++++++++----
.../rewrite/PushDownTopNDistinctThroughUnion.java | 11 ++++++++-
.../rules/rewrite/PushDownTopNThroughJoin.java | 16 ++++++++++---
.../rules/rewrite/PushDownTopNThroughUnion.java | 12 ++++++++--
.../rules/rewrite/PushDownTopNThroughWindow.java | 11 +++++++++
.../rewrite/PushDownVectorTopNIntoOlapScan.java | 7 ++++++
.../doris/nereids/rules/rewrite/SplitLimit.java | 9 +++++++-
.../nereids/trees/plans/logical/LogicalLimit.java | 6 +++++
.../nereids/trees/plans/logical/LogicalTopN.java | 5 ++++
.../trees/plans/physical/PhysicalLimit.java | 10 ++++++++
.../nereids/trees/plans/physical/PhysicalTopN.java | 5 ++++
.../java/org/apache/doris/nereids/util/Utils.java | 16 +++++++++++++
.../org/apache/doris/nereids/util/UtilsTest.java | 17 ++++++++++++++
.../push_down_top_n_through_union.groovy | 27 ++++++++++++++++++++++
24 files changed, 239 insertions(+), 17 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 1c037c65edb..002cec00050 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -2609,7 +2609,13 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
// push sort to scan opt
if (sortNode.getChild(0) instanceof OlapScanNode) {
OlapScanNode scanNode = ((OlapScanNode) sortNode.getChild(0));
- if (checkPushSort(sortNode, scanNode.getOlapTable())) {
+ // When the offset is set, the pushed scan limit is limit +
offset. If that overflows the
+ // long range it would wrap to a negative limit, so skip the
whole push-sort-to-scan
+ // optimization (do not push the sort info either); the sort
node still applies the real
+ // limit/offset. Sort info and sort limit are always pushed
together.
+ boolean limitOverflows = sortNode.getOffset() > 0
+ && Utils.addOverflows(sortNode.getLimit(),
sortNode.getOffset());
+ if (checkPushSort(sortNode, scanNode.getOlapTable()) &&
!limitOverflows) {
SortInfo sortInfo = sortNode.getSortInfo();
scanNode.setSortInfo(sortInfo);
if (sortNode.getOffset() > 0) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushTopnToAgg.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushTopnToAgg.java
index 44bbfacc794..768a15143e2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushTopnToAgg.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushTopnToAgg.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.TopnP
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
import org.apache.doris.nereids.trees.plans.physical.PhysicalTopN;
import org.apache.doris.nereids.util.AggregateUtils;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
/**
@@ -49,6 +50,11 @@ public class PushTopnToAgg extends PlanPostProcessor {
@Override
public Plan visitPhysicalTopN(PhysicalTopN<? extends Plan> topN,
CascadesContext ctx) {
topN.child().accept(this, ctx);
+ if (Utils.addOverflows(topN.getLimit(), topN.getOffset())) {
+ // limit + offset overflows the long range: no aggregate can hold
that many rows, so
+ // pushing a topn limit into the aggregate cannot reduce anything;
leave the topn as is.
+ return topN;
+ }
if (ConnectContext.get().getSessionVariable().topnOptLimitThreshold <=
topN.getLimit() + topN.getOffset()
&& !ConnectContext.get().getSessionVariable().pushTopnToAgg) {
return topN;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalTopNToPhysicalTopN.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalTopNToPhysicalTopN.java
index 2371407555b..359e4b0bf9c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalTopNToPhysicalTopN.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalTopNToPhysicalTopN.java
@@ -17,12 +17,14 @@
package org.apache.doris.nereids.rules.implementation;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.SortPhase;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.trees.plans.physical.PhysicalTopN;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.Lists;
@@ -46,13 +48,20 @@ public class LogicalTopNToPhysicalTopN extends
OneImplementationRuleFactory {
* mergeTopN(limit, off, require gather) -> localTopN(off+limit, 0,
require any)
*/
private List<PhysicalTopN<? extends Plan>> twoPhaseSort(LogicalTopN<?
extends Plan> logicalTopN) {
- PhysicalTopN<Plan> localSort = new
PhysicalTopN<>(logicalTopN.getOrderKeys(),
- logicalTopN.getLimit() + logicalTopN.getOffset(), 0,
SortPhase.LOCAL_SORT,
- logicalTopN.getLogicalProperties(), logicalTopN.child(0));
int sortPhaseNum = 0;
if (ConnectContext.get() != null) {
sortPhaseNum =
ConnectContext.get().getSessionVariable().sortPhaseNum;
}
+ // The local sort keeps limit + offset rows. When that overflows the
long range (e.g. LIMIT
+ // and OFFSET both BIGINT_MAX), we cannot produce a valid physical
TopN: the two-phase
+ // MERGE_SORT needs a local limit that overflows, and even a
single-phase GATHER_SORT would
+ // pass the overflowing limit/offset to BE where HeapSorter also
computes limit + offset.
+ if (Utils.addOverflows(logicalTopN.getLimit(),
logicalTopN.getOffset())) {
+ throw new AnalysisException("limit + offset overflows the long
range");
+ }
+ PhysicalTopN<Plan> localSort = new
PhysicalTopN<>(logicalTopN.getOrderKeys(),
+ logicalTopN.getLimit() + logicalTopN.getOffset(), 0,
SortPhase.LOCAL_SORT,
+ logicalTopN.getLogicalProperties(), logicalTopN.child(0));
if (sortPhaseNum == 1) {
PhysicalTopN<Plan> onePhaseSort = new
PhysicalTopN<>(logicalTopN.getOrderKeys(), logicalTopN.getLimit(),
logicalTopN.getOffset(), SortPhase.GATHER_SORT,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CollectLimitAboveConsumer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CollectLimitAboveConsumer.java
index bb573304e3a..0b0009fcdaf 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CollectLimitAboveConsumer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CollectLimitAboveConsumer.java
@@ -23,6 +23,7 @@ import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
@@ -52,7 +53,15 @@ public class CollectLimitAboveConsumer implements
RewriteRuleFactory {
private void collectLimitRows(CascadesContext cascadesContext,
LogicalLimit<?> limit,
LogicalCTEConsumer cteConsumer) {
- cascadesContext.putConsumerIdToLimitRows(
- cteConsumer.getRelationId(), limit.getLimit() +
limit.getOffset());
+ // The recorded value is the number of rows this consumer needs (limit
+ offset). When that
+ // overflows the long range the consumer effectively needs all rows,
so do not record anything:
+ // tryToConstructLimit treats a missing entry as "unbounded" and
leaves the producer unlimited
+ // (this consumer's own limit still applies above it). Recording a
wrapped-around negative count
+ // would corrupt the producer row bound.
+ if (Utils.addOverflows(limit.getLimit(), limit.getOffset())) {
+ return;
+ }
+ cascadesContext.putConsumerIdToLimitRows(cteConsumer.getRelationId(),
+ limit.getLimit() + limit.getOffset());
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LimitAggToTopNAgg.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LimitAggToTopNAgg.java
index f2ca5e68817..a209a1dd2da 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LimitAggToTopNAgg.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LimitAggToTopNAgg.java
@@ -30,6 +30,7 @@ import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -57,6 +58,7 @@ public class LimitAggToTopNAgg implements RewriteRuleFactory {
logicalLimit(logicalAggregate())
.when(limit -> ConnectContext.get() != null
&&
ConnectContext.get().getSessionVariable().pushTopnToAgg
+ && !Utils.addOverflows(limit.getLimit(),
limit.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= limit.getLimit() + limit.getOffset())
.when(limit -> {
@@ -73,6 +75,7 @@ public class LimitAggToTopNAgg implements RewriteRuleFactory {
logicalLimit(logicalProject(logicalAggregate()))
.when(limit -> ConnectContext.get() != null
&&
ConnectContext.get().getSessionVariable().pushTopnToAgg
+ && !Utils.addOverflows(limit.getLimit(),
limit.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= limit.getLimit() + limit.getOffset())
.when(limit -> {
@@ -93,6 +96,7 @@ public class LimitAggToTopNAgg implements RewriteRuleFactory {
logicalTopN(logicalAggregate())
.when(topn -> ConnectContext.get() != null
&&
ConnectContext.get().getSessionVariable().pushTopnToAgg
+ && !Utils.addOverflows(topn.getLimit(),
topn.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= topn.getLimit() + topn.getOffset())
.when(topn -> {
@@ -115,6 +119,7 @@ public class LimitAggToTopNAgg implements
RewriteRuleFactory {
logicalTopN(logicalProject(logicalAggregate()))
.when(topn -> ConnectContext.get() != null
&&
ConnectContext.get().getSessionVariable().pushTopnToAgg
+ && !Utils.addOverflows(topn.getLimit(),
topn.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= topn.getLimit() + topn.getOffset())
.when(topn -> {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeLimits.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeLimits.java
index e544eed07ce..e9e466123b0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeLimits.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeLimits.java
@@ -17,10 +17,12 @@
package org.apache.doris.nereids.rules.rewrite;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
+import org.apache.doris.nereids.util.Utils;
/**
* This rule aims to merge consecutive limits.
@@ -54,7 +56,16 @@ public class MergeLimits extends OneRewriteRuleFactory {
}).toRule(RuleType.MERGE_LIMITS);
}
+ /**
+ * Merge two consecutive limits' offsets into a single offset. Consecutive
limits must be merged,
+ * and an overflowing combined offset cannot be represented as a single
offset, so fail fast
+ * instead of wrapping to a negative offset.
+ */
public static long mergeOffset(long upperOffset, long bottomOffset) {
+ if (Utils.addOverflows(upperOffset, bottomOffset)) {
+ throw new AnalysisException(
+ "offset overflows long range when merging limits: " +
upperOffset + " + " + bottomOffset);
+ }
return upperOffset + bottomOffset;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
index 27ef659118a..f4549ea9bdd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
@@ -17,11 +17,13 @@
package org.apache.doris.nereids.rules.rewrite;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.properties.OrderKey;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.util.Utils;
import java.util.List;
@@ -57,6 +59,10 @@ public class MergeTopNs extends OneRewriteRuleFactory {
long limit = topN.getLimit();
long childOffset = childTopN.getOffset();
long childLimit = childTopN.getLimit();
+ if (Utils.addOverflows(offset, childOffset)) {
+ throw new AnalysisException(
+ "offset overflows long range when merging
TopNs: " + offset + " + " + childOffset);
+ }
long newOffset = offset + childOffset;
// The parent's offset is applied on top of the child's
output, so only
// (childLimit - offset) of the child's rows survive.
Clamp the merged limit
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughJoin.java
index 906007decf4..960e0e0b128 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughJoin.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughJoin.java
@@ -26,6 +26,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -46,6 +47,7 @@ public class PushDownLimitDistinctThroughJoin implements
RewriteRuleFactory {
.when(LogicalAggregate::isDistinct))
.when(limit ->
ConnectContext.get() != null
+ &&
!Utils.addOverflows(limit.getLimit(), limit.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= limit.getLimit() +
limit.getOffset())
.then(limit -> {
@@ -64,6 +66,12 @@ public class PushDownLimitDistinctThroughJoin implements
RewriteRuleFactory {
logicalLimit(logicalAggregate(logicalProject(logicalJoin()).when(LogicalProject::isAllSlots))
.when(LogicalAggregate::isDistinct))
.then(limit -> {
+ // limit + offset overflowing means no child can
hold that many rows, so the
+ // push-down cannot reduce anything; skip it. (The
direct branch is gated the
+ // same way via topn_opt_limit_threshold.)
+ if (Utils.addOverflows(limit.getLimit(),
limit.getOffset())) {
+ return null;
+ }
LogicalAggregate<LogicalProject<LogicalJoin<Plan,
Plan>>> agg = limit.child();
LogicalProject<LogicalJoin<Plan, Plan>> project =
agg.child();
LogicalJoin<Plan, Plan> join = project.child();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughUnion.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughUnion.java
index 1a136673a1d..fd75ca129cc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughUnion.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownLimitDistinctThroughUnion.java
@@ -27,6 +27,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
@@ -64,6 +65,12 @@ public class PushDownLimitDistinctThroughUnion implements
RewriteRuleFactory {
logicalLimit(logicalAggregate(logicalUnion().when(union ->
union.getQualifier() == Qualifier.ALL))
.when(agg -> agg.isDistinct()))
.then(limit -> {
+ // limit + offset overflowing the long range means
no child can hold that
+ // many rows, so pushing the limit below the union
cannot reduce anything;
+ // skip the rewrite. The parent limit still
applies the original limit/offset.
+ if (Utils.addOverflows(limit.getLimit(),
limit.getOffset())) {
+ return null;
+ }
LogicalAggregate<LogicalUnion> agg = limit.child();
LogicalUnion union = agg.child();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownScoreTopNIntoOlapScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownScoreTopNIntoOlapScan.java
index d24a6018438..1395c2febb0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownScoreTopNIntoOlapScan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownScoreTopNIntoOlapScan.java
@@ -44,6 +44,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.thrift.TExprOpcode;
import com.google.common.collect.ImmutableList;
@@ -188,6 +189,12 @@ public class PushDownScoreTopNIntoOlapScan implements
RewriteRuleFactory {
+ " for push down optimization");
}
+ // When limit + offset overflows the long range, the pushed scan limit
would wrap to a
+ // negative value; skip the push-down and let the TopN above the scan
apply limit/offset.
+ if (Utils.addOverflows(topN.getLimit(), topN.getOffset())) {
+ return null;
+ }
+
// All conditions met, perform the push down.
// This is the core action: push score() as a virtual column and also
push the
// topN info.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
index 4b0b8a89dc4..a67f06decd7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -49,6 +50,7 @@ public class PushDownTopNDistinctThroughJoin implements
RewriteRuleFactory {
// TODO: complex order by
.when(topn ->
ConnectContext.get() != null
+ &&
!Utils.addOverflows(topn.getLimit(), topn.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= topn.getLimit() + topn.getOffset())
.when(topN ->
topN.getOrderKeys().stream().map(OrderKey::getExpr)
@@ -70,6 +72,13 @@ public class PushDownTopNDistinctThroughJoin implements
RewriteRuleFactory {
.when(topN ->
topN.getOrderKeys().stream().map(OrderKey::getExpr)
.allMatch(Slot.class::isInstance))
.then(topN -> {
+ // limit + offset overflowing the long range means
no child can hold that
+ // many rows, so pushing the TopN below the join
cannot reduce anything;
+ // skip the rewrite. (The direct branch is gated
the same way via
+ // topn_opt_limit_threshold.)
+ if (Utils.addOverflows(topN.getLimit(),
topN.getOffset())) {
+ return null;
+ }
LogicalAggregate<LogicalProject<LogicalJoin<Plan,
Plan>>> distinct = topN.child();
LogicalProject<LogicalJoin<Plan, Plan>> project =
distinct.child();
LogicalJoin<Plan, Plan> join = project.child();
@@ -95,6 +104,7 @@ public class PushDownTopNDistinctThroughJoin implements
RewriteRuleFactory {
}
private Plan pushTopNThroughJoin(LogicalTopN<? extends Plan> topN,
LogicalJoin<Plan, Plan> join) {
+ long childLimit = topN.getLimit() + topN.getOffset();
Set<Slot> groupBySlots = ((LogicalAggregate<?>)
topN.child()).getGroupByExpressions().stream()
.flatMap(e ->
e.getInputSlots().stream()).collect(Collectors.toSet());
switch (join.getJoinType()) {
@@ -107,7 +117,7 @@ public class PushDownTopNDistinctThroughJoin implements
RewriteRuleFactory {
join.left().getOutputSet(), topN.getOrderKeys());
if (!pushedOrderKeys.isEmpty()) {
LogicalTopN<Plan> left = topN.withLimitOrderKeyAndChild(
- topN.getLimit() + topN.getOffset(), 0,
pushedOrderKeys,
+ childLimit, 0, pushedOrderKeys,
PlanUtils.distinct(join.left()));
return join.withChildren(left, join.right());
}
@@ -122,7 +132,7 @@ public class PushDownTopNDistinctThroughJoin implements
RewriteRuleFactory {
join.right().getOutputSet(), topN.getOrderKeys());
if (!pushedOrderKeys.isEmpty()) {
LogicalTopN<Plan> right = topN.withLimitOrderKeyAndChild(
- topN.getLimit() + topN.getOffset(), 0,
pushedOrderKeys,
+ childLimit, 0, pushedOrderKeys,
PlanUtils.distinct(join.right()));
return join.withChildren(join.left(), right);
}
@@ -135,14 +145,14 @@ public class PushDownTopNDistinctThroughJoin implements
RewriteRuleFactory {
join.left().getOutputSet(), topN.getOrderKeys());
if (!(join.left() instanceof TopN) &&
!leftPushedOrderKeys.isEmpty()) {
leftChild = topN.withLimitOrderKeyAndChild(
- topN.getLimit() + topN.getOffset(), 0,
leftPushedOrderKeys,
+ childLimit, 0, leftPushedOrderKeys,
PlanUtils.distinct(join.left()));
}
List<OrderKey> rightPushedOrderKeys =
getPushedOrderKeys(groupBySlots,
join.right().getOutputSet(), topN.getOrderKeys());
if (!(join.right() instanceof TopN) &&
!rightPushedOrderKeys.isEmpty()) {
rightChild = topN.withLimitOrderKeyAndChild(
- topN.getLimit() + topN.getOffset(), 0,
rightPushedOrderKeys,
+ childLimit, 0, rightPushedOrderKeys,
PlanUtils.distinct(join.right()));
}
if (leftChild == join.left() && rightChild == join.right()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
index cde42c7ad24..3e0dee97aca 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
@@ -29,6 +29,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
@@ -65,6 +66,14 @@ public class PushDownTopNDistinctThroughUnion implements
RewriteRuleFactory {
logicalTopN(logicalAggregate(logicalUnion().when(union ->
union.getQualifier() == Qualifier.ALL))
.when(agg -> agg.isDistinct()))
.then(topN -> {
+ // The child TopN must keep limit + offset rows.
When that sum overflows the
+ // long range (e.g. LIMIT and OFFSET both
BIGINT_MAX) no child can hold that many
+ // rows, so pushing down cannot reduce anything
and would create an illegal
+ // negative limit. Skip the rewrite; the parent
TopN still applies limit/offset.
+ if (Utils.addOverflows(topN.getLimit(),
topN.getOffset())) {
+ return null;
+ }
+ long childLimit = topN.getLimit() +
topN.getOffset();
LogicalAggregate<LogicalUnion> agg = topN.child();
LogicalUnion union = agg.child();
List<Plan> newChildren = new ArrayList<>();
@@ -78,7 +87,7 @@ public class PushDownTopNDistinctThroughUnion implements
RewriteRuleFactory {
.map(orderKey ->
orderKey.withExpression(
ExpressionUtils.replace(orderKey.getExpr(), replaceMap)))
.collect(ImmutableList.toImmutableList());
- newChildren.add(new LogicalTopN<>(orderKeys,
topN.getLimit() + topN.getOffset(), 0,
+ newChildren.add(new LogicalTopN<>(orderKeys,
childLimit, 0,
PlanUtils.distinct(union.child(childIdx))));
}
if (union.children().equals(newChildren)) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughJoin.java
index 14d56ad9172..69dfffd5af3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughJoin.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughJoin.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -51,6 +52,7 @@ public class PushDownTopNThroughJoin implements
RewriteRuleFactory {
// TODO: complex orderby
.when(topn ->
ConnectContext.get() != null
+ &&
!Utils.addOverflows(topn.getLimit(), topn.getOffset())
&&
ConnectContext.get().getSessionVariable().topnOptLimitThreshold
>= topn.getLimit() + topn.getOffset())
.when(topN ->
topN.getOrderKeys().stream().map(OrderKey::getExpr)
@@ -70,6 +72,13 @@ public class PushDownTopNThroughJoin implements
RewriteRuleFactory {
.when(topN ->
topN.getOrderKeys().stream().map(OrderKey::getExpr)
.allMatch(Slot.class::isInstance))
.then(topN -> {
+ // limit + offset overflowing the long range means
no child can hold that
+ // many rows, so pushing the TopN below the join
cannot reduce anything;
+ // skip the rewrite. (The direct TopN -> Join
branch is gated the same way
+ // via topn_opt_limit_threshold.)
+ if (Utils.addOverflows(topN.getLimit(),
topN.getOffset())) {
+ return null;
+ }
LogicalProject<LogicalJoin<Plan, Plan>> project =
topN.child();
LogicalJoin<Plan, Plan> join = project.child();
@@ -183,6 +192,7 @@ public class PushDownTopNThroughJoin implements
RewriteRuleFactory {
private Plan pushTopNToChild(LogicalTopN<? extends Plan> topN, Plan child,
List<Slot> orderbySlots,
List<Slot> requiredOutputSlots) {
+ long childLimit = topN.getLimit() + topN.getOffset();
// Keep only the slots that this child actually outputs. When pushing
to
// one side of an outer join, requiredOutputSlots may include columns
from
// the other side which this child cannot provide.
@@ -191,19 +201,19 @@ public class PushDownTopNThroughJoin implements
RewriteRuleFactory {
.filter(childOutputSet::contains)
.collect(Collectors.toList());
if (childRequired.isEmpty()) {
- return topN.withLimitChild(topN.getLimit() + topN.getOffset(), 0,
child);
+ return topN.withLimitChild(childLimit, 0, child);
}
// Ensure the child outputs all required columns. If the child already
// outputs everything needed, no extra Project is required.
Set<Slot> requiredSet = new HashSet<>(childRequired);
requiredSet.addAll(orderbySlots);
if (childOutputSet.containsAll(requiredSet)) {
- return topN.withLimitChild(topN.getLimit() + topN.getOffset(), 0,
child);
+ return topN.withLimitChild(childLimit, 0, child);
}
List<NamedExpression> childOutput = child.getOutput().stream()
.filter(requiredSet::contains)
.collect(Collectors.toList());
Plan topNChild = PlanUtils.projectOrSelf(childOutput, child);
- return topN.withLimitChild(topN.getLimit() + topN.getOffset(), 0,
topNChild);
+ return topN.withLimitChild(childLimit, 0, topNChild);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughUnion.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughUnion.java
index b02d7a1e45d..0416a4c5a79 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughUnion.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughUnion.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
@@ -62,6 +63,14 @@ public class PushDownTopNThroughUnion implements
RewriteRuleFactory {
return ImmutableList.of(
logicalTopN(logicalUnion().when(union -> union.getQualifier()
== Qualifier.ALL))
.then(topN -> {
+ // The child TopN must keep limit + offset rows.
When that sum overflows the
+ // long range (e.g. LIMIT and OFFSET both
BIGINT_MAX) no child can hold that many
+ // rows, so pushing down cannot reduce anything
and would create an illegal
+ // negative limit. Skip the rewrite; the parent
TopN still applies limit/offset.
+ if (Utils.addOverflows(topN.getLimit(),
topN.getOffset())) {
+ return null;
+ }
+ long childLimit = topN.getLimit() +
topN.getOffset();
LogicalUnion union = topN.child();
List<Plan> newChildren = new ArrayList<>();
for (int j = 0; j < union.children().size(); j++) {
@@ -77,8 +86,7 @@ public class PushDownTopNThroughUnion implements
RewriteRuleFactory {
.map(orderKey ->
orderKey.withExpression(
ExpressionUtils.replace(orderKey.getExpr(), replaceMap)))
.collect(ImmutableList.toImmutableList());
- newChildren.add(
- new LogicalTopN<>(orderKeys,
topN.getLimit() + topN.getOffset(), 0, child));
+ newChildren.add(new LogicalTopN<>(orderKeys,
childLimit, 0, child));
}
if (union.children().equals(newChildren)) {
return null;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughWindow.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughWindow.java
index 8dc4cd6f73a..6c5117448bf 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughWindow.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNThroughWindow.java
@@ -30,6 +30,7 @@ import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
@@ -53,6 +54,11 @@ public class PushDownTopNThroughWindow implements
RewriteRuleFactory {
if (!checkTopNForPartitionLimitPushDown(topn, windowExprId)) {
return topn;
}
+ // When limit + offset overflows the long range (e.g. LIMIT
and OFFSET both
+ // BIGINT_MAX) the partition limit cannot reduce anything;
skip the push-down.
+ if (Utils.addOverflows(topn.getLimit(), topn.getOffset())) {
+ return topn;
+ }
long partitionLimit = topn.getLimit() + topn.getOffset();
Pair<WindowExpression, Long> windowFuncLongPair = window
.getPushDownWindowFuncAndLimit(null, partitionLimit);
@@ -76,6 +82,11 @@ public class PushDownTopNThroughWindow implements
RewriteRuleFactory {
if (!checkTopNForPartitionLimitPushDown(topn, windowExprId)) {
return topn;
}
+ // When limit + offset overflows the long range (e.g. LIMIT
and OFFSET both
+ // BIGINT_MAX) the partition limit cannot reduce anything;
skip the push-down.
+ if (Utils.addOverflows(topn.getLimit(), topn.getOffset())) {
+ return topn;
+ }
long partitionLimit = topn.getLimit() + topn.getOffset();
Pair<WindowExpression, Long> windowFuncLongPair = window
.getPushDownWindowFuncAndLimit(null, partitionLimit);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
index 6d449b3f51c..597f331e28d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
@@ -36,6 +36,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Maps;
@@ -146,6 +147,12 @@ public class PushDownVectorTopNIntoOlapScan implements
RewriteRuleFactory {
return null;
}
+ // When limit + offset overflows the long range, the pushed scan limit
would wrap to a
+ // negative value; skip the push-down and let the TopN above the scan
apply limit/offset.
+ if (Utils.addOverflows(topN.getLimit(), topN.getOffset())) {
+ return null;
+ }
+
Plan plan = scan.appendVirtualColumnsAndTopN(
ImmutableList.of(orderKeyAlias),
topN.getOrderKeys(), Optional.of(topN.getLimit() +
topN.getOffset()),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SplitLimit.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SplitLimit.java
index 06f7760c433..0db8a0380d0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SplitLimit.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SplitLimit.java
@@ -17,10 +17,12 @@
package org.apache.doris.nereids.rules.rewrite;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.plans.LimitPhase;
import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
+import org.apache.doris.nereids.util.Utils;
/**
* Split limit into two phase
@@ -38,8 +40,13 @@ public class SplitLimit extends OneRewriteRuleFactory {
.then(limit -> {
long l = limit.getLimit();
long o = limit.getOffset();
+ if (Utils.addOverflows(l, o)) {
+ throw new AnalysisException(
+ "limit + offset overflows the long range in
two-phase limit");
+ }
return new LogicalLimit<>(l, o,
- LimitPhase.GLOBAL, new LogicalLimit<>(l + o, 0,
LimitPhase.LOCAL, limit.child())
+ LimitPhase.GLOBAL,
+ new LogicalLimit<>(l + o, 0, LimitPhase.LOCAL,
limit.child())
);
}).toRule(RuleType.SPLIT_LIMIT);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java
index ca418e361cc..e92d396bbf8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java
@@ -55,9 +55,15 @@ public class LogicalLimit<CHILD_TYPE extends Plan> extends
LogicalUnary<CHILD_TY
this(limit, offset, phase, Optional.empty(), Optional.empty(), child);
}
+ /** Build a LogicalLimit with the given limit, offset and phase. */
public LogicalLimit(long limit, long offset, LimitPhase phase,
Optional<GroupExpression> groupExpression,
Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
super(PlanType.LOGICAL_LIMIT, groupExpression, logicalProperties,
child);
+ // limit/offset are always non-negative ("no limit" is represented by
Long.MAX_VALUE). A
+ // negative value here means limit + offset overflowed somewhere
upstream and would produce an
+ // illegal plan that hangs in BE; fail fast instead.
+ Preconditions.checkArgument(limit >= 0 && offset >= 0,
+ "LogicalLimit limit and offset must be non-negative, but got
limit=%s, offset=%s", limit, offset);
this.limit = limit;
this.offset = offset;
this.phase = phase;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTopN.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTopN.java
index bc685158b57..c1d7f37060f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTopN.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTopN.java
@@ -61,6 +61,11 @@ public class LogicalTopN<CHILD_TYPE extends Plan> extends
LogicalUnary<CHILD_TYP
Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
super(PlanType.LOGICAL_TOP_N, groupExpression, logicalProperties,
child);
this.orderKeys =
ImmutableList.copyOf(Objects.requireNonNull(orderKeys, "orderKeys can not be
null"));
+ // limit/offset are always non-negative ("no limit" is represented by
Long.MAX_VALUE). A
+ // negative value here means limit + offset overflowed somewhere
upstream and would produce an
+ // illegal plan that hangs in BE; fail fast instead.
+ Preconditions.checkArgument(limit >= 0 && offset >= 0,
+ "LogicalTopN limit and offset must be non-negative, but got
limit=%s, offset=%s", limit, offset);
this.limit = limit;
this.offset = offset;
this.expressions = Suppliers.memoize(() -> {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLimit.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLimit.java
index 61babb6de6b..2ced8023e91 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLimit.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalLimit.java
@@ -64,6 +64,11 @@ public class PhysicalLimit<CHILD_TYPE extends Plan> extends
PhysicalUnary<CHILD_
LimitPhase phase, Optional<GroupExpression> groupExpression,
LogicalProperties logicalProperties,
CHILD_TYPE child) {
super(PlanType.PHYSICAL_LIMIT, groupExpression, logicalProperties,
child);
+ // limit/offset are always non-negative ("no limit" is represented by
Long.MAX_VALUE). A
+ // negative value here means limit + offset overflowed somewhere
upstream and would produce an
+ // illegal plan that hangs in BE; fail fast instead.
+ Preconditions.checkArgument(limit >= 0 && offset >= 0,
+ "PhysicalLimit limit and offset must be non-negative, but got
limit=%s, offset=%s", limit, offset);
this.limit = limit;
this.offset = offset;
this.phase = phase;
@@ -82,6 +87,11 @@ public class PhysicalLimit<CHILD_TYPE extends Plan> extends
PhysicalUnary<CHILD_
Statistics statistics, CHILD_TYPE child) {
super(PlanType.PHYSICAL_LIMIT, groupExpression, logicalProperties,
physicalProperties, statistics,
child);
+ // limit/offset are always non-negative ("no limit" is represented by
Long.MAX_VALUE). A
+ // negative value here means limit + offset overflowed somewhere
upstream and would produce an
+ // illegal plan that hangs in BE; fail fast instead.
+ Preconditions.checkArgument(limit >= 0 && offset >= 0,
+ "PhysicalLimit limit and offset must be non-negative, but got
limit=%s, offset=%s", limit, offset);
this.limit = limit;
this.offset = offset;
this.phase = phase;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
index 1d6ff2f28d9..5868a3a48b2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
@@ -71,6 +71,11 @@ public class PhysicalTopN<CHILD_TYPE extends Plan> extends
AbstractPhysicalSort<
super(PlanType.PHYSICAL_TOP_N, orderKeys, phase, groupExpression,
logicalProperties, physicalProperties,
statistics, child);
Objects.requireNonNull(orderKeys, "orderKeys should not be null in
PhysicalTopN.");
+ // limit/offset are always non-negative ("no limit" is represented by
Long.MAX_VALUE). A
+ // negative value here means limit + offset overflowed somewhere
upstream and would produce an
+ // illegal plan that hangs in BE; fail fast instead.
+ Preconditions.checkArgument(limit >= 0 && offset >= 0,
+ "PhysicalTopN limit and offset must be non-negative, but got
limit=%s, offset=%s", limit, offset);
this.limit = limit;
this.offset = offset;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
index e01ae2baf60..c268f01f73c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
@@ -130,6 +130,22 @@ public class Utils {
return false;
}
+ /**
+ * Whether {@code a + b} overflows the signed long range. {@code a} and
{@code b} are expected to
+ * be non-negative (e.g. a limit and an offset).
+ *
+ * <p>When combining a {@code limit} and an {@code offset} into the number
of rows a child must
+ * keep (e.g. pushing a TopN/Limit down, or building a two-phase sort),
the raw {@code limit +
+ * offset} can exceed the long range when both are close to {@code
BIGINT_MAX}; it then wraps to a
+ * negative number, which is an illegal limit and produces a broken plan.
Overflow here means the
+ * child would need more rows than any relation can hold (no relation
exceeds {@code
+ * Long.MAX_VALUE} rows), so the optimization cannot reduce anything:
callers should skip the
+ * rewrite and fall back to the correct unoptimized execution.
+ */
+ public static boolean addOverflows(long a, long b) {
+ return a > Long.MAX_VALUE - b;
+ }
+
/**
* Wrapper to a function without return value.
*/
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/UtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/UtilsTest.java
index 0c7d903311f..f1fcccab3e7 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/UtilsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/UtilsTest.java
@@ -32,4 +32,21 @@ public class UtilsTest {
String en = "database123";
Assertions.assertFalse(Utils.containChinese(en));
}
+
+ @Test
+ public void testAddOverflows() {
+ // sums that fit in the long range do not overflow
+ Assertions.assertFalse(Utils.addOverflows(0, 0));
+ Assertions.assertFalse(Utils.addOverflows(1, 2));
+ Assertions.assertFalse(Utils.addOverflows(Long.MAX_VALUE, 0));
+ Assertions.assertFalse(Utils.addOverflows(0, Long.MAX_VALUE));
+ Assertions.assertFalse(Utils.addOverflows(Long.MAX_VALUE - 1, 1));
+
+ // limit + offset that exceeds the long range overflows. This is the
case (e.g. LIMIT and
+ // OFFSET both BIGINT_MAX) that previously produced a negative,
illegal child limit when
+ // pushing TopN/Limit down; callers now skip the rewrite instead.
+ Assertions.assertTrue(Utils.addOverflows(Long.MAX_VALUE, 1));
+ Assertions.assertTrue(Utils.addOverflows(Long.MAX_VALUE,
Long.MAX_VALUE));
+ Assertions.assertTrue(Utils.addOverflows(Long.MAX_VALUE - 1, 2));
+ }
}
diff --git
a/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_through_union.groovy
b/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_through_union.groovy
index 28ef3d1b61a..cb2c819cd1f 100644
---
a/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_through_union.groovy
+++
b/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_through_union.groovy
@@ -312,4 +312,31 @@ suite("push_down_top_n_through_union") {
ORDER BY logTimestamp desc
LIMIT 8;
"""
+
+ // When limit + offset overflows the long range (e.g. LIMIT and OFFSET
both at BIGINT_MAX),
+ // the planner reports an error instead of silently producing a broken
plan. Both the TopN path
+ // (ORDER BY + LIMIT/OFFSET) and the Limit path (LIMIT/OFFSET without
ORDER BY) are covered.
+ test {
+ sql """
+ select count(*) as c from (
+ select id from (
+ select 1 as id union all select 2 as id union all select 3
as id
+ ) t
+ order by id limit 9223372036854775807 offset
9223372036854775807
+ ) s;
+ """
+ exception "limit + offset overflows"
+ }
+
+ test {
+ sql """
+ select count(*) as c from (
+ select id from (
+ select 1 as id union all select 2 as id union all select 3
as id
+ ) t
+ limit 9223372036854775807 offset 9223372036854775807
+ ) s;
+ """
+ exception "limit + offset overflows"
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
