This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7060c1cb1cd [opt](ivm) Reduce snapshot checks for outer join NULL
repair (#70561)
7060c1cb1cd is described below
commit 7060c1cb1cdfdc7b8063728df5a0604b508bc9f4
Author: yujun <[email protected]>
AuthorDate: Thu Nov 19 14:44:25 2026 +0800
[opt](ivm) Reduce snapshot checks for outer join NULL repair (#70561)
### Background
During incremental refresh, a change to B in `A LEFT JOIN B` can change
a joined row and can also retract or insert the NULL-extended A row. The
same repair is needed in the opposite direction for RIGHT and FULL
joins. FULL joins additionally preserve unmatched rows from the delta
itself.
### What problem does this PR solve?
Issue Number: None
Related PR: #70630 (sign-aware sequence for repair rows)
PR Type: Performance
Outer join refresh builds separate pre- and post-snapshot existence
checks for NULL repair. If a retained A row has at most one matching B
row in either snapshot, its repair sign is exactly the negative change
in its match count. This PR computes that change from the signed delta
and also shares key-based repair events between LEFT, RIGHT and FULL
equijoins.
### Details
Each row describes one contribution from the changing input B. A is the
opposite-side snapshot. The LEFT/RIGHT column covers changes from the
null-producing side. When the preserved input changes, that contribution
uses the original directional outer join directly. For FULL, apply the
column once for each changing input.
| ON condition and B uniqueness | LEFT/RIGHT, B changes | FULL, B
changes | B pre/post snapshots for repair |
| --- | --- | --- | --- |
| Pure deterministic ordinary equalities. All B equality keys support
value-preserving grouping, and B's direct equality slots cover a unique
key of the original relation. | `A INNER JOIN events`. Events are B
detail plus signed key repairs. | `A RIGHT OUTER JOIN B_delta` for
detail, including unmatched B rows, plus `A INNER JOIN
signed_key_repairs`. | Neither |
| Pure deterministic ordinary equalities. All B equality keys support
value-preserving grouping, and B's uniqueness is unproven, for example
with a partial composite key or an expression key. | `A INNER JOIN
events`. Events are B detail plus pre/post key `ANTI JOIN` repairs. | `A
RIGHT OUTER JOIN B_delta` for detail plus `A INNER JOIN` to the union of
the pre/post key repairs. | Both |
| Deterministic equality plus a residual predicate. B's direct equality
slots cover a unique key, A is repeatable, and A's ON inputs support
value-preserving grouping. | `A INNER JOIN B_delta` for detail plus
signed repair grouped by A's ON inputs. | `A RIGHT OUTER JOIN B_delta`
for detail plus the same repair grouped by A's ON inputs. | Neither |
| All remaining cases. Examples include a `FLOAT` or `DOUBLE` B equality
key, an unproven B key with a residual predicate, and a retained-side
producer or input type that cannot be used for tuple repair. | `A INNER
JOIN B_delta` for detail plus the two pre/post `SEMI JOIN` / `ANTI JOIN`
repairs. | `A RIGHT OUTER JOIN B_delta` for detail plus the same two
pre/post repairs. | Both |
Both pure-equality event paths group B keys before probing A. `GROUP BY`
can merge `+0.0` and `-0.0` while a hash join distinguishes them. An
update between signed zeros can cancel distinct repair events or make
the grouped event probe the wrong rows. Keys that do not support
value-preserving grouping use the original per-row SEMI/ANTI snapshot
repair and still refresh incrementally.
**Pure equijoin, LEFT/RIGHT, unique B with eligible equality keys.** The
key aggregate replaces two repair inputs below the same probe. Ordinary
same-key payload updates still flow through detail.
```diff
INNER JOIN (complete join key)
|-- A_post snapshot
`-- UNION ALL
|-- delta_B detail events
- |-- PROJECT (NULL payload, factor = -1)
- | `-- LEFT ANTI JOIN
- | |-- FILTER (has_positive)
- | | `-- AGGREGATE (key, MAX positive/negative flags)
- | | `-- delta_B
- | `-- B_pre snapshot
- `-- PROJECT (NULL payload, factor = +1)
- `-- LEFT ANTI JOIN
- |-- FILTER (has_negative)
- | `-- AGGREGATE (key, MAX positive/negative flags)
- | `-- delta_B
- `-- B_post snapshot
+ `-- PROJECT (NULL payload, factor = -net_count)
+ `-- FILTER (net_count <> 0)
+ `-- AGGREGATE (key, SUM(dml_factor) AS net_count)
+ `-- FILTER (join keys IS NOT NULL)
+ `-- delta_B
```
**Pure equijoin, FULL, B delta with eligible equality keys.** The detail
branch emits even a B row with no A match. Only repair events probe A by
key.
```diff
UNION ALL
|-- A_snapshot RIGHT OUTER JOIN B_delta
-|-- (A_snapshot SEMI JOIN B_inserts) ANTI JOIN B_pre
-`-- (A_snapshot SEMI JOIN B_deletes) ANTI JOIN B_post
+`-- A_snapshot INNER JOIN repair_events ON key
+ unique B: FILTER net <> 0 -> AGG BY key, SUM(B_delta.factor)
+ unproven B: UNION ALL
+ |-- affected insert keys ANTI JOIN B_pre
+ `-- affected delete keys ANTI JOIN B_post
```
**Additional ON predicate, unique B.** For example, given `A.k = B.k AND
A.threshold < B.amount`, a same-key update can change the match for only
some A rows. Evaluate the entire predicate before counting. Retained
rows with the same values for every A slot in ON have identical matches.
For n such rows, the signed sum is n times the match change, so its sign
determines one unit repair per original row. Look up the retained bag
with NULL-safe comparisons on the complete tuple. This also works for
nullable, colliding or independently generated row IDs. The tuple path
accepts booleans, integral and decimal values, character strings,
dates/timestamps, IPs and UUIDs. Other input types use snapshot repair.
The same fallback applies when a retained subtree produces
nondeterministic values, including an otherwise deterministic-looking ON
slot backed by a UUID alias.
```diff
UNION ALL
|-- detail (INNER for LEFT/RIGHT null-side delta; directional OUTER for
FULL)
-|-- (A_snapshot SEMI JOIN B_inserts ON full predicate) ANTI JOIN B_pre
-`-- (A_snapshot SEMI JOIN B_deletes ON full predicate) ANTI JOIN B_post
+`-- PROJECT A payload, NULL B payload, factor = -sign(net)
+ `-- A_snapshot INNER JOIN changed_tuples ON each A predicate input <=>
its group key
+ `-- FILTER net <> 0
+ `-- AGG BY A.k, A.threshold, SUM(B_delta.factor) AS net
+ `-- A_snapshot INNER JOIN B_delta ON full predicate
```
**Results and cost.** For a unique B that qualifies for signed repair,
both repair snapshot reads disappear. In FULL equijoins with eligible
equality keys, the opposite-side snapshot occurs twice instead of three
times. The residual path evaluates the match join for repair as well as
detail and looks up the retained rows for each changed tuple. These are
logical plan differences. Runtime will depend on the changed rows,
selectivity and snapshot sizes. No elapsed-time benchmark was run.
---
.../apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java | 344 +++++++++++++--------
.../apache/doris/nereids/types/BooleanType.java | 5 +
.../org/apache/doris/nereids/types/DataType.java | 11 +
.../apache/doris/nereids/types/DecimalV2Type.java | 5 +
.../apache/doris/nereids/types/DecimalV3Type.java | 5 +
.../org/apache/doris/nereids/types/IPv4Type.java | 5 +
.../org/apache/doris/nereids/types/IPv6Type.java | 5 +
.../org/apache/doris/nereids/types/UuidType.java | 5 +
.../nereids/types/coercion/CharacterType.java | 5 +
.../doris/nereids/types/coercion/DateLikeType.java | 5 +
.../doris/nereids/types/coercion/IntegralType.java | 5 +
.../apache/doris/mtmv/ivm/IvmDeltaTestBase.java | 6 +-
.../doris/mtmv/ivm/IvmJoinDeltaHandlerTest.java | 274 +++++++++++++++-
.../ivm/test_ivm_unique_key_null_repair.out | 219 +++++++++++++
.../mtmv_p0/ivm/test_ivm_explain_refresh.groovy | 12 +-
.../ivm/test_ivm_unique_key_null_repair.groovy | 303 ++++++++++++++++++
16 files changed, 1073 insertions(+), 141 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java
index 5f6b992be1e..9bef6949df2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java
@@ -21,15 +21,22 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.common.Pair;
import org.apache.doris.nereids.rules.exploration.join.JoinReorderContext;
import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.IsNull;
import org.apache.doris.nereids.trees.expressions.LessThan;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Not;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.Subtract;
import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.BigIntLiteral;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
import org.apache.doris.nereids.trees.plans.JoinType;
@@ -39,6 +46,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
+import org.apache.doris.nereids.types.TinyIntType;
import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.JoinUtils;
@@ -65,10 +73,34 @@ import java.util.Set;
* <li>delta side is the null side: emit joined rows, plus repair rows for
old/new null-side MV rows</li>
* </ul>
*
- * <p>For FULL OUTER JOIN both physical sides retain unmatched rows. A left
delta is therefore handled as
- * {@code left_delta LEFT OUTER JOIN right_snapshot} plus repair rows for
right-only dangling rows; a right delta is
- * symmetric. Those repair rows are modeled with a branch-local retained side
and null side instead of a global
- * retained/null-side assumption.
+ * <p>For each contribution below, B is the changing input and A is the
opposite snapshot. In LEFT/RIGHT
+ * joins these cases apply when B is the null-producing side. A delta on the
preserved side uses the
+ * original directional outer join directly. For FULL joins either side can
play B.
+ *
+ * <pre>
+ * Case 1. Pure deterministic equality. B's keys support value-preserving
grouping
+ * and the equality slots prove B unique.
+ * LEFT/RIGHT, B changes. Join A once to the union of B detail and signed
key repair events.
+ * FULL, B changes. Keep unmatched B rows in outer join detail. Join A
separately to signed key repairs.
+ *
+ * Case 2. Pure deterministic equality. B's keys support value-preserving
grouping
+ * and B's uniqueness is unproven.
+ * LEFT/RIGHT, B changes. Join A once to the union of B detail and two
pre/post key ANTI events.
+ * FULL, B changes. Keep unmatched B rows in outer join detail. Join A to
the two key ANTI events.
+ *
+ * Case 3. Deterministic equality plus a residual predicate. The equality
slots prove B unique,
+ * A is repeatable and A's ON inputs support value-preserving grouping.
+ * LEFT/RIGHT, B changes. Inner join detail plus signed repair per A
predicate tuple.
+ * FULL, B changes. Directional outer join detail plus the same tuple repair.
+ *
+ * Case 4. All remaining conditions, including equality keys that cannot be
grouped safely.
+ * LEFT/RIGHT, B changes. Inner join detail plus pre/post SEMI/ANTI repair.
+ * FULL, B changes. Directional outer join detail plus the same SEMI/ANTI
repair.
+ * </pre>
+ *
+ * <p>Equality here means extractable, ordinary equality keys. Uniqueness must
hold for direct B slots
+ * on the original relation. A signed repair emits the negative change in
match count, since a unique
+ * key limits each A row to zero or one match in each snapshot.
*/
class IvmJoinDeltaHandler {
@@ -77,6 +109,7 @@ class IvmJoinDeltaHandler {
private static final String NULL_SIDE_DETAIL_DELTA_ALIAS =
"__DORIS_IVM_NULL_SIDE_DETAIL_DELTA__";
private static final String NULL_SIDE_EVENT_KEY_PREFIX =
"__DORIS_IVM_NULL_SIDE_EVENT_KEY_";
private static final String NULL_SIDE_KEY_DELTA_ALIAS =
"__DORIS_IVM_NULL_SIDE_KEY_DELTA__";
+ private static final String NULL_SIDE_KEY_NET_COUNT_ALIAS =
"__DORIS_IVM_NULL_SIDE_NET_COUNT__";
private static final String NULL_SIDE_KEY_POSITIVE_ALIAS =
"__DORIS_IVM_NULL_SIDE_KEY_POSITIVE__";
private static final String NULL_SIDE_KEY_NEGATIVE_ALIAS =
"__DORIS_IVM_NULL_SIDE_KEY_NEGATIVE__";
private static final String NULL_SIDE_PRE_SNAPSHOT_ALIAS =
"__DORIS_IVM_NULL_SIDE_PRE_SNAPSHOT__";
@@ -146,9 +179,7 @@ class IvmJoinDeltaHandler {
}
Plan deltaSideChild = deltaOnLeft ? join.left() : join.right();
return rewriteNullSideDelta(new NullSideDeltaContext(join,
deltaOnLeft, delta, leftInput, rightInput,
- helper.preSnapshot(deltaSideChild, rewriteState),
- helper.postSnapshot(deltaSideChild, rewriteState),
- rewriteState, delta.maxDeltaIndex));
+ deltaSideChild, rewriteState, delta.maxDeltaIndex));
}
private IvmDeltaRewriteResult rewriteInnerJoinContribution(LogicalJoin<?
extends Plan, ? extends Plan> join,
@@ -213,58 +244,37 @@ class IvmJoinDeltaHandler {
/**
* Delta from the null side may change both joined rows and null-side rows.
*
- * <p>LEFT/RIGHT OUTER JOIN null-side delta can use the optimized event
path for pure deterministic equality
- * predicates. FULL OUTER JOIN uses the general three-branch path because
the first branch must also keep
- * unmatched delta-side rows with LEFT/RIGHT OUTER JOIN.
+ * <p>LEFT/RIGHT OUTER JOIN null-side delta can use the event path for
pure deterministic equality with
+ * value-preserving keys. FULL OUTER JOIN keeps unmatched delta rows in a
separate outer-join branch
+ * and probes only the repair events.
*/
private IvmDeltaRewriteResult rewriteNullSideDelta(NullSideDeltaContext
deltaContext) {
LogicalJoin<? extends Plan, ? extends Plan> join = deltaContext.join;
- boolean isFullOuterJoin = join.getJoinType().isFullOuterJoin();
- EquiJoinKeys equiJoinKeys = isFullOuterJoin ? null :
extractEquiJoinKeys(join);
- if (equiJoinKeys != null) {
+ EquiJoinKeys equiJoinKeys = extractEquiJoinKeys(join);
+ // Both event paths group the delta keys before probing the retained
side. Grouping can merge
+ // floating-point signed zeros that a hash join distinguishes.
+ if (equiJoinKeys != null && !equiJoinKeys.hasResidual
+ && deltaContext.deltaSideKeyExpressions(equiJoinKeys).stream()
+ .allMatch(key ->
key.getDataType().supportsValuePreservingGrouping())) {
return rewriteNullSideDeltaWithNullSideEvents(deltaContext,
equiJoinKeys);
- } else {
- return rewriteNullSideDeltaWithRepairBranches(deltaContext);
}
+ if (equiJoinKeys != null && equiJoinKeys.hasResidual) {
+ Set<Slot> retainedInputs = deltaContext.isDeltaOnLeft
+ ? join.getRightConditionSlot() :
join.getLeftConditionSlot();
+ if (hasUniqueNullSideKeys(deltaContext, equiJoinKeys)
+ && retainedInputs.stream().allMatch(slot ->
slot.getDataType().supportsValuePreservingGrouping())
+ && isRepeatableRetainedInput(deltaContext)) {
+ return combineWithJoinedRows(deltaContext,
+ ImmutableList.of(buildMatchedTupleRepair(deltaContext,
retainedInputs)));
+ }
+ }
+ return combineWithJoinedRows(deltaContext,
buildNullSideRepairProjects(deltaContext));
}
- /**
- * General null-side delta rewrite using one joined-row branch and two
NULL-row repair branches.
- */
- private IvmDeltaRewriteResult
rewriteNullSideDeltaWithRepairBranches(NullSideDeltaContext deltaContext) {
- // Null-side delta for:
- // retained_snapshot OUTER JOIN null_side_delta
- //
- // It has three parts:
- // 1. Joined rows:
- // retained_snapshot INNER JOIN null_side_delta
- // or for FULL OUTER JOIN:
- // left_delta LEFT OUTER JOIN right_snapshot
- // left_snapshot RIGHT OUTER JOIN right_delta
- // FULL OUTER JOIN uses LEFT/RIGHT OUTER JOIN here to keep
unmatched delta-side rows, and skips the
- // null-side event path because that path probes
retained_snapshot with INNER JOIN.
- //
- // 2. Remove old null-side rows when null-side inserts create the
first match:
- // retained_snapshot LEFT SEMI JOIN null_side_insert_delta
- // LEFT ANTI JOIN null_side_pre_snapshot
- // The semi join keeps retained-side rows affected by this delta
without
- // multiplying them by matched delta rows. The anti join then
keeps only rows
- // that had no matching null-side row before this delta. For
those rows, the old MV
- // contained one row with null-side columns filled as NULL, so we
emit that
- // null-side row with dml_factor = -1.
- //
- // 3. Add new null-side rows when null-side deletes remove the last
match:
- // retained_snapshot LEFT SEMI JOIN null_side_delete_delta
- // LEFT ANTI JOIN null_side_post_snapshot
- // The semi join keeps retained-side rows affected by this delta
without
- // multiplying them by matched delta rows. The anti join then
keeps only rows
- // that have no matching null-side row after this delta. For
those rows, the new MV
- // needs one null-side row, so we emit that row with dml_factor =
+1.
+ private IvmDeltaRewriteResult combineWithJoinedRows(NullSideDeltaContext
deltaContext, List<Plan> repairs) {
Plan joinedProject = rewriteNullSideJoinedRowsDelta(deltaContext);
- List<Plan> repairProjects = buildNullSideRepairProjects(deltaContext);
-
- LogicalUnion union = helper.buildUnionAll(ImmutableList.of(
- joinedProject, repairProjects.get(0), repairProjects.get(1)));
+ LogicalUnion union =
helper.buildUnionAll(ImmutableList.<Plan>builder().add(joinedProject)
+ .addAll(repairs).build());
LogicalProject<Plan> outputProject = helper.projectUnionOutputs(union,
joinedProject.getOutput());
Slot dmlFactor = findSlotByName(outputProject.getOutput(),
Column.IVM_DML_FACTOR_COL);
Slot sequence = findSlotByName(outputProject.getOutput(),
Column.SEQUENCE_COL);
@@ -298,6 +308,50 @@ class IvmJoinDeltaHandler {
deltaContext.deltaSideResult().maxDeltaIndex),
slotMapping).plan;
}
+ private boolean isRepeatableRetainedInput(NullSideDeltaContext context) {
+ // ON may reference an alias such as UUID() AS tag. Its slot looks
deterministic, but the
+ // match aggregate and the payload lookup evaluate the retained
subtree independently.
+ Plan retained = context.isDeltaOnLeft ? context.join.right() :
context.join.left();
+ return !retained.anyMatch(node -> ((Plan)
node).getExpressions().stream()
+ .anyMatch(Expression::containsNondeterministic));
+ }
+
+ /**
+ * Retained rows with identical ON inputs match exactly the same delta
rows. For n such rows
+ * the signed sum is n * (post - pre), where each endpoint is zero or one
by delta-side uniqueness.
+ * Its sign gives one repair per original retained row, independent of
multiplicity or row-id.
+ */
+ private Plan buildMatchedTupleRepair(NullSideDeltaContext context,
Set<Slot> retainedInputs) {
+ Pair<Plan, Map<Slot, Slot>> retained =
freshSideInput(context.nonDeltaSideInput());
+ Pair<Plan, Map<Slot, Slot>> delta =
helper.freshPlan(context.deltaSideResult().plan);
+ Map<Slot, Slot> mapping = new HashMap<>(retained.second);
+ // The delta contribution was rebound to the source output ExprIds
before rewriting this join.
+ mapping.putAll(delta.second);
+ LogicalJoin<Plan, Plan> matches = new
LogicalJoin<>(JoinType.INNER_JOIN,
+ ExpressionUtils.replace(context.join.getHashJoinConjuncts(),
mapping),
+ ExpressionUtils.replace(context.join.getOtherJoinConjuncts(),
mapping),
+ context.join.getDistributeHint(), retained.first, delta.first,
JoinReorderContext.EMPTY);
+ Pair<Plan, Map<Slot, Slot>> payload =
freshSideInput(context.nonDeltaSideInput());
+ ImmutableList.Builder<Slot> groupBy = ImmutableList.builder();
+ ImmutableList.Builder<Expression> lookup = ImmutableList.builder();
+ for (Slot input : retainedInputs) {
+ Slot key = retained.second.get(input);
+ groupBy.add(key);
+ // A residual such as COALESCE(A.threshold, 0) can match when an
input is NULL.
+ lookup.add(new NullSafeEqual(payload.second.get(input), key));
+ }
+ List<Slot> tuple = groupBy.build();
+ Alias netCount = new Alias(new
Sum(delta.second.get(context.deltaSideResult().dmlFactorSlot)),
+ NULL_SIDE_KEY_NET_COUNT_ALIAS);
+ Plan aggregate = new LogicalAggregate<>(ImmutableList.copyOf(tuple),
+
ImmutableList.<NamedExpression>builder().addAll(tuple).add(netCount).build(),
matches);
+ Plan changedRows = new LogicalJoin<>(JoinType.INNER_JOIN,
lookup.build(), payload.first,
+ changedNetCounts(aggregate, netCount),
JoinReorderContext.EMPTY);
+ Expression factor = new If(new GreaterThan(netCount.toSlot(), new
BigIntLiteral(0)),
+ new TinyIntLiteral((byte) -1), new TinyIntLiteral((byte) 1));
+ return projectNullSideRepairOutputs(context, changedRows, factor,
payload.second);
+ }
+
/**
* Build the two NULL-row repair projects for null-side delta.
*
@@ -327,9 +381,9 @@ class IvmJoinDeltaHandler {
// full null-side relation, so retain all branches and only replace
the one delta scan
// with its pre/post snapshot.
Pair<Plan, Map<Slot, Slot>> nullSidePreSnapshot =
helper.remapOutputs(helper.aliasPlan(
- freshSnapshot(deltaContext.deltaSidePreSnapshot),
NULL_SIDE_PRE_SNAPSHOT_ALIAS));
+ freshSnapshot(deltaContext.deltaSideSnapshot(false)),
NULL_SIDE_PRE_SNAPSHOT_ALIAS));
Pair<Plan, Map<Slot, Slot>> nullSidePostSnapshot =
helper.remapOutputs(helper.aliasPlan(
- freshSnapshot(deltaContext.deltaSidePostSnapshot),
NULL_SIDE_POST_SNAPSHOT_ALIAS));
+ freshSnapshot(deltaContext.deltaSideSnapshot(true)),
NULL_SIDE_POST_SNAPSHOT_ALIAS));
LogicalProject<Plan> preNullProject =
buildNullSideRepairProject(deltaContext,
freshSideInput(deltaContext.nonDeltaSideInput()),
insertedNullSideDelta.second, nullSideInserts,
nullSidePreSnapshot,
@@ -347,50 +401,12 @@ class IvmJoinDeltaHandler {
private IvmDeltaRewriteResult rewriteNullSideDeltaWithNullSideEvents(
NullSideDeltaContext deltaContext, EquiJoinKeys equiJoinKeys) {
LogicalJoin<? extends Plan, ? extends Plan> join = deltaContext.join;
- // Null-side delta for equi outer join can be reduced to one probe:
- // retained_snapshot INNER JOIN null_side_events
- //
- // The important point is that null_side_events is not just
"null_side_delta with
- // another name". It encodes every MV row change caused by the null
side
- // into rows that can be joined by key with the retained-side snapshot.
- // After that encoding, retained_snapshot does not need to know
whether an event
- // is a real null-side row change or a NULL-row repair; it only probes
once by
- // the event key and projects the event payload.
- //
- // null_side_events has three parts:
- // 1. Detail null-side delta rows:
- // null_side_delta
- // These rows keep the original null-side outputs and null-side
delta dml_factor,
- // so the final join emits normal joined row changes.
- //
- // 2. Remove old null-side rows when null-side inserts create the
first match:
- // affected null_side_insert_delta keys LEFT ANTI JOIN
null_side_pre_snapshot
- // For those keys, the old MV contained one row with null-side
columns filled
- // as NULL. The event carries the join keys, sets null-side
outputs to NULL,
- // and uses dml_factor = -1.
- //
- // 3. Add new null-side rows when null-side deletes remove the last
match:
- // affected null_side_delete_delta keys LEFT ANTI JOIN
null_side_post_snapshot
- // For those keys, the new MV needs one null-side row. The event
carries
- // the join keys, sets null-side outputs to NULL, and uses
dml_factor = +1.
- //
- // By merging the bare join and NULL-row repair rows into
null_side_events, the
- // retained-side snapshot is scanned/probed once instead of three
times.
- //
- // This requires pure deterministic equi keys. Expressions like
- // f(left_slots) = g(right_slots)
- // are supported, because the null-side event relation can materialize
- // the null-side key as event_key and the final probe can evaluate
- // the retained-side key against event_key. Conditions such as
- // left.k = right.k AND left.v > right.v
- // are not supported here, because the null side alone cannot decide
which
- // retained-side rows are affected by the non-hash predicate. Such
joins fall back to
- // the three repair branches. Unique functions such as random()/uuid()
are
- // also rejected before this path, because recomputing them in
different
- // event branches would produce unstable keys.
+ // A one-sided outer join probes detail and repair events together.
FULL must also emit
+ // unmatched delta rows, so its detail branch uses an outer join and
only repairs use this probe.
+ boolean fullOuter = join.getJoinType().isFullOuterJoin();
List<Slot> targetOutputs = buildOuterDeltaOutputs(deltaContext);
Pair<Plan, Map<Slot, Slot>> retainedSnapshot =
freshSideInput(deltaContext.nonDeltaSideInput());
- NullSideEventPlan nullSideEvents =
buildNullSideEventPlan(deltaContext, equiJoinKeys);
+ NullSideEventPlan nullSideEvents =
buildNullSideEventPlan(deltaContext, equiJoinKeys, fullOuter);
// Join retained rows with the event relation by the extracted
equality keys. A detail event produces a
// normal joined-row change; a repair event produces the same retained
row with null-side payloads set to NULL.
@@ -407,7 +423,10 @@ class IvmJoinDeltaHandler {
retainedSnapshot.first, nullSideEvents.plan,
JoinReorderContext.EMPTY);
LogicalProject<Plan> outputProject =
projectEventJoinOutputs(targetOutputs,
eventJoin, retainedSnapshot.second,
nullSideEvents.nullSideOutputMapping,
- nullSideEvents.dmlFactorSlot, nullSideEvents.sequenceSlot);
+ nullSideEvents.dmlFactorSlot, nullSideEvents.sequenceSlot,
fullOuter);
+ if (fullOuter) {
+ return combineWithJoinedRows(deltaContext,
ImmutableList.of(outputProject));
+ }
Slot dmlFactor = findSlotByName(outputProject.getOutput(),
Column.IVM_DML_FACTOR_COL);
Slot sequence = findSlotByName(outputProject.getOutput(),
Column.SEQUENCE_COL);
return new IvmDeltaRewriteResult(outputProject, dmlFactor, sequence,
@@ -511,15 +530,25 @@ class IvmJoinDeltaHandler {
* retained snapshot after probing by the event keys.
*/
private NullSideEventPlan buildNullSideEventPlan(
- NullSideDeltaContext deltaContext, EquiJoinKeys equiJoinKeys) {
- Plan detailEvent = buildNullSideDetailEvent(deltaContext,
equiJoinKeys);
- Plan preNullEvent = buildNullSideRepairEvent(deltaContext,
equiJoinKeys, false,
- new TinyIntLiteral((byte) -1));
- Plan postNullEvent = buildNullSideRepairEvent(deltaContext,
equiJoinKeys, true,
- new TinyIntLiteral((byte) 1));
- LogicalUnion union =
helper.buildUnionAll(ImmutableList.of(detailEvent, preNullEvent,
postNullEvent));
+ NullSideDeltaContext deltaContext, EquiJoinKeys equiJoinKeys,
boolean repairsOnly) {
+ ImmutableList.Builder<Plan> events = ImmutableList.builder();
+ if (!repairsOnly) {
+ events.add(buildNullSideDetailEvent(deltaContext, equiJoinKeys));
+ }
+ if (hasUniqueNullSideKeys(deltaContext, equiJoinKeys)) {
+ events.add(buildUniqueKeyRepairEvent(deltaContext, equiJoinKeys));
+ } else {
+ Plan preNullEvent = buildNullSideRepairEvent(deltaContext,
equiJoinKeys, false,
+ new TinyIntLiteral((byte) -1));
+ Plan postNullEvent = buildNullSideRepairEvent(deltaContext,
equiJoinKeys, true,
+ new TinyIntLiteral((byte) 1));
+ events.add(preNullEvent);
+ events.add(postNullEvent);
+ }
+ List<Plan> branches = events.build();
+ Plan eventPlan = branches.size() == 1 ? branches.get(0) :
helper.buildUnionAll(branches);
- List<Slot> unionOutputs = union.getOutput();
+ List<Slot> unionOutputs = eventPlan.getOutput();
Map<Slot, Slot> nullSideOutputMapping = new HashMap<>();
int nullSideOutputStart =
deltaContext.deltaSideKeyExpressions(equiJoinKeys).size();
int nullSideOutputIndex = 0;
@@ -530,10 +559,63 @@ class IvmJoinDeltaHandler {
List<Slot> eventKeySlots = unionOutputs.subList(0,
deltaContext.deltaSideKeyExpressions(equiJoinKeys).size());
Slot dmlFactorSlot = unionOutputs.get(unionOutputs.size() - 2);
Slot sequenceSlot = unionOutputs.get(unionOutputs.size() - 1);
- return new NullSideEventPlan(union, nullSideOutputMapping,
eventKeySlots,
+ return new NullSideEventPlan(eventPlan, nullSideOutputMapping,
eventKeySlots,
dmlFactorSlot, sequenceSlot);
}
+ /**
+ * Use the snapshot relation's proof, never the incremental stream: a
stream can contain several versions of
+ * the same key. Only direct slots are accepted; a non-injective
expression over a unique slot need not be unique.
+ */
+ private boolean hasUniqueNullSideKeys(NullSideDeltaContext context,
EquiJoinKeys keys) {
+ Set<Slot> keySlots = new HashSet<>();
+ for (Expression key : context.deltaSideKeyExpressions(keys)) {
+ if (!(key instanceof Slot)) {
+ return false;
+ }
+ keySlots.add((Slot) key);
+ }
+ Plan nullSide = context.isDeltaOnLeft ? context.join.left() :
context.join.right();
+ return nullSide.getLogicalProperties().getTrait().isUnique(keySlots);
+ }
+
+ /**
+ * For a unique matching key, pre/post multiplicities are in {0, 1}. Hence
the NULL-row change is exactly
+ * -(post - pre), without querying either snapshot. Detail changes remain
in their own UNION branch.
+ */
+ private Plan buildUniqueKeyRepairEvent(NullSideDeltaContext context,
EquiJoinKeys keys) {
+ Pair<Plan, Map<Slot, Slot>> delta =
helper.remapOutputs(helper.aliasPlan(
+ helper.freshPlan(context.deltaSideResult().plan),
NULL_SIDE_KEY_DELTA_ALIAS));
+ ImmutableList.Builder<Expression> groupBy = ImmutableList.builder();
+ ImmutableList.Builder<NamedExpression> outputs =
ImmutableList.builder();
+ ImmutableSet.Builder<Expression> nonNullKeys = ImmutableSet.builder();
+ List<Expression> keyExpressions =
context.deltaSideKeyExpressions(keys);
+ for (int i = 0; i < keyExpressions.size(); i++) {
+ Expression key = ExpressionUtils.replace(keyExpressions.get(i),
delta.second);
+ groupBy.add(key);
+ outputs.add(new Alias(key, eventKeyName(i)));
+ // Ordinary equality cannot match NULL keys. Filter only repair
candidates, never detail events.
+ nonNullKeys.add(new Not(new IsNull(key)));
+ }
+ Alias netCount = new Alias(new
Sum(delta.second.get(context.deltaSideResult().dmlFactorSlot)),
+ NULL_SIDE_KEY_NET_COUNT_ALIAS);
+ outputs.add(netCount);
+ Plan aggregate = new LogicalAggregate<>(groupBy.build(),
outputs.build(),
+ new LogicalFilter<>(nonNullKeys.build(), delta.first));
+ return projectNullSideRepairEvent(context, changedNetCounts(aggregate,
netCount),
+ aggregate.getOutput().subList(0, keyExpressions.size()),
repairFactor(netCount));
+ }
+
+ private Plan changedNetCounts(Plan aggregate, Alias netCount) {
+ return new LogicalFilter<>(ImmutableSet.of(new Not(new
EqualTo(netCount.toSlot(), new BigIntLiteral(0)))),
+ aggregate);
+ }
+
+ private Expression repairFactor(Alias netCount) {
+ // Unique endpoints constrain the matched-row count change to -1/0/+1.
+ return new Cast(new Subtract(new BigIntLiteral(0), netCount.toSlot()),
TinyIntType.INSTANCE);
+ }
+
/**
* Build detail events from raw null-side delta rows. These events produce
normal joined-row changes after
* probing the retained-side snapshot.
@@ -573,8 +655,7 @@ class IvmJoinDeltaHandler {
new GreaterThan(flagSlot, new TinyIntLiteral((byte) 0))),
deltaKeys.plan);
String snapshotAlias = postSnapshot ? NULL_SIDE_POST_SNAPSHOT_ALIAS :
NULL_SIDE_PRE_SNAPSHOT_ALIAS;
Pair<Plan, Map<Slot, Slot>> nullSideSnapshot =
helper.remapOutputs(helper.aliasPlan(
- freshSnapshot(postSnapshot ? deltaContext.deltaSidePostSnapshot
- : deltaContext.deltaSidePreSnapshot), snapshotAlias));
+ freshSnapshot(deltaContext.deltaSideSnapshot(postSnapshot)),
snapshotAlias));
ImmutableList.Builder<Expression> antiConjuncts =
ImmutableList.builderWithExpectedSize(
deltaContext.deltaSideKeyExpressions(equiJoinKeys).size());
@@ -587,16 +668,21 @@ class IvmJoinDeltaHandler {
antiConjuncts.build(), ImmutableList.of(),
join.getDistributeHint(),
affectedKeys, nullSideSnapshot.first,
JoinReorderContext.EMPTY);
+ return projectNullSideRepairEvent(deltaContext, antiJoin,
deltaKeys.keySlots, dmlFactor);
+ }
+
+ private Plan projectNullSideRepairEvent(NullSideDeltaContext deltaContext,
Plan source,
+ List<Slot> keySlots, Expression dmlFactor) {
ImmutableList.Builder<NamedExpression> projects =
ImmutableList.builder();
- for (int i = 0; i < deltaKeys.keySlots.size(); i++) {
- projects.add(new Alias(deltaKeys.keySlots.get(i),
eventKeyName(i)));
+ for (int i = 0; i < keySlots.size(); i++) {
+ projects.add(new Alias(keySlots.get(i), eventKeyName(i)));
}
for (Slot slot : nullSideValueSlots(deltaContext)) {
projects.add(new Alias(new NullLiteral(slot.getDataType()),
slot.getName()));
}
projects.add(new Alias(dmlFactor, Column.IVM_DML_FACTOR_COL));
projects.add(new Alias(deltaContext.paddingSequence(dmlFactor),
Column.SEQUENCE_COL));
- return new LogicalProject<>(projects.build(), antiJoin);
+ return new LogicalProject<>(projects.build(), source);
}
/**
@@ -635,7 +721,7 @@ class IvmJoinDeltaHandler {
*/
private LogicalProject<Plan> projectEventJoinOutputs(List<Slot>
targetOutputs, Plan source,
Map<Slot, Slot> retainedOutputMapping, Map<Slot, Slot>
nullSideOutputMapping,
- Slot dmlFactorSlot, Slot sequenceSlot) {
+ Slot dmlFactorSlot, Slot sequenceSlot, boolean freshOutputs) {
ImmutableList.Builder<NamedExpression> projects =
ImmutableList.builderWithExpectedSize(
targetOutputs.size());
for (Slot target : targetOutputs) {
@@ -654,7 +740,8 @@ class IvmJoinDeltaHandler {
throw new IvmException(IvmFailureReason.PLAN_REWRITE_FAILED,
"IVM outer join event rewrite lost output slot: " +
target);
}
- projects.add(new Alias(target.getExprId(), expr,
target.getName()));
+ projects.add(freshOutputs ? new Alias(expr, target.getName()) :
new Alias(target.getExprId(), expr,
+ target.getName()));
}
return new LogicalProject<>(projects.build(), source);
}
@@ -677,10 +764,9 @@ class IvmJoinDeltaHandler {
}
/**
- * Extract pure equi-join keys from both hash conjuncts and other
conjuncts.
+ * Extract deterministic equi-join keys from both hash conjuncts and other
conjuncts.
*
- * Return null when there is no hashable equality, or when any residual
non-hash condition remains. The null
- * result makes the null-side rewrite fall back to the general
repair-branch path.
+ * Record residual predicates separately: their truth can depend on the
individual retained row.
*
* This intentionally accepts expression keys, not only slot-to-slot keys.
For example,
* date_trunc(left.dt) = date_trunc(right.dt)
@@ -695,12 +781,12 @@ class IvmJoinDeltaHandler {
.addAll(join.getHashJoinConjuncts())
.addAll(join.getOtherJoinConjuncts())
.build();
- if (conjuncts.isEmpty()) {
+ if (conjuncts.stream().anyMatch(Expression::containsNondeterministic))
{
return null;
}
Pair<List<Expression>, List<Expression>> extractedConjuncts =
JoinUtils.extractExpressionForHashTable(
join.left().getOutput(), join.right().getOutput(), conjuncts);
- if (extractedConjuncts.first.isEmpty() ||
!extractedConjuncts.second.isEmpty()) {
+ if (extractedConjuncts.first.isEmpty()) {
return null;
}
Set<ExprId> leftExprIds = outputExprIds(join.left());
@@ -714,9 +800,6 @@ class IvmJoinDeltaHandler {
EqualTo equalTo = (EqualTo) conjunct;
Expression left = equalTo.left();
Expression right = equalTo.right();
- if (left.containsVolatileExpression() ||
right.containsVolatileExpression()) {
- return null;
- }
if (isBoundBy(left, leftExprIds) && isBoundBy(right,
rightExprIds)) {
leftKeys.add(left);
rightKeys.add(right);
@@ -727,7 +810,7 @@ class IvmJoinDeltaHandler {
return null;
}
}
- return new EquiJoinKeys(leftKeys.build(), rightKeys.build());
+ return new EquiJoinKeys(leftKeys.build(), rightKeys.build(),
!extractedConjuncts.second.isEmpty());
}
/**
@@ -804,13 +887,15 @@ class IvmJoinDeltaHandler {
private static class EquiJoinKeys {
private final List<Expression> leftExpressions;
private final List<Expression> rightExpressions;
+ private final boolean hasResidual;
/**
* Store key expressions in physical join-child order.
*/
- private EquiJoinKeys(List<Expression> leftExpressions,
List<Expression> rightExpressions) {
+ private EquiJoinKeys(List<Expression> leftExpressions,
List<Expression> rightExpressions, boolean hasResidual) {
this.leftExpressions = leftExpressions;
this.rightExpressions = rightExpressions;
+ this.hasResidual = hasResidual;
}
}
@@ -828,8 +913,7 @@ class IvmJoinDeltaHandler {
private final IvmDeltaRewriteResult deltaSideResult;
private final SideInput leftInput;
private final SideInput rightInput;
- private final Pair<Plan, Map<Slot, Slot>> deltaSidePreSnapshot;
- private final Pair<Plan, Map<Slot, Slot>> deltaSidePostSnapshot;
+ private final Plan deltaSideSource;
private final IvmDeltaRewriteState rewriteState;
private final int maxDeltaIndex;
@@ -839,16 +923,14 @@ class IvmJoinDeltaHandler {
private NullSideDeltaContext(LogicalJoin<? extends Plan, ? extends
Plan> join,
boolean deltaOnLeft, IvmDeltaRewriteResult deltaSideResult,
SideInput leftInput, SideInput rightInput,
- Pair<Plan, Map<Slot, Slot>> deltaSidePreSnapshot,
- Pair<Plan, Map<Slot, Slot>> deltaSidePostSnapshot,
IvmDeltaRewriteState rewriteState,
+ Plan deltaSideSource, IvmDeltaRewriteState rewriteState,
int maxDeltaIndex) {
this.join = join;
this.isDeltaOnLeft = deltaOnLeft;
this.deltaSideResult = deltaSideResult;
this.leftInput = leftInput;
this.rightInput = rightInput;
- this.deltaSidePreSnapshot = deltaSidePreSnapshot;
- this.deltaSidePostSnapshot = deltaSidePostSnapshot;
+ this.deltaSideSource = deltaSideSource;
this.rewriteState = rewriteState;
this.maxDeltaIndex = maxDeltaIndex;
}
@@ -860,6 +942,12 @@ class IvmJoinDeltaHandler {
return deltaSideResult;
}
+ // Unique-key event repair does not need either null-side snapshot.
+ private Pair<Plan, Map<Slot, Slot>> deltaSideSnapshot(boolean
postSnapshot) {
+ return postSnapshot ?
IvmDeltaRewriteHelper.INSTANCE.postSnapshot(deltaSideSource, rewriteState)
+ :
IvmDeltaRewriteHelper.INSTANCE.preSnapshot(deltaSideSource, rewriteState);
+ }
+
private Expression paddingSequence(Expression dmlFactor) {
// A joined subtree can emit cancelling positive/negative rows
absent from both snapshots.
// Both NULL-row repairs then target the same row-id; keep the
positive repair last.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/BooleanType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/BooleanType.java
index 4cff03adf29..e082fdb811f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/BooleanType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/BooleanType.java
@@ -35,6 +35,11 @@ public class BooleanType extends PrimitiveType {
private BooleanType() {
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
if (target instanceof DecimalV2Type) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
index 929d4313f54..9dc87cb4076 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
@@ -873,6 +873,17 @@ public abstract class DataType {
return this.equals(target);
}
+ /**
+ * Whether grouping and null-safe equality can be used to replace a value
by its group
+ * representative in deterministic expressions. Both operations must
identify the same
+ * values, and expressions must observe the same value for all members of
a group.
+ * For example, {@code signbit(CAST('+0.0' AS DOUBLE))} is false and
+ * {@code signbit(CAST('-0.0' AS DOUBLE))} is true, even though both zeros
belong to the same group.
+ */
+ public boolean supportsValuePreservingGrouping() {
+ return false;
+ }
+
public static List<DataType> trivialTypes() {
return Type.getTrivialTypes()
.stream()
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV2Type.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV2Type.java
index 9c651f9f2ea..15aad43d050 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV2Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV2Type.java
@@ -167,6 +167,11 @@ public class DecimalV2Type extends FractionalType {
return DecimalV2Type.createDecimalV2Type(range + scale, scale);
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
// DECIMALV2 is a deprecated legacy type. Its declared precision and
scale do not always
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV3Type.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV3Type.java
index f31b9826599..0ddca068785 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV3Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV3Type.java
@@ -240,6 +240,11 @@ public class DecimalV3Type extends FractionalType {
}
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
if (target instanceof DecimalV2Type) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv4Type.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv4Type.java
index 543a138da15..db4652d03c4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv4Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv4Type.java
@@ -37,6 +37,11 @@ public class IPv4Type extends PrimitiveType {
private IPv4Type() {
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
// BE maps all 32 IPv4 bits into IPv6 and prints a canonical address
for string casts.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv6Type.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv6Type.java
index 729f83bfe55..86c90cc51cf 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv6Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/IPv6Type.java
@@ -37,6 +37,11 @@ public class IPv6Type extends PrimitiveType {
private IPv6Type() {
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
// IPv6Value::to_string emits a canonical textual representation of
all 128 address bits.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/UuidType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/UuidType.java
index c12a0bc802e..0be68b78b68 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/UuidType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/UuidType.java
@@ -37,6 +37,11 @@ public class UuidType extends PrimitiveType {
return Type.UUID;
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean acceptsType(DataType other) {
return other instanceof UuidType;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
index c799766a510..6ccb0a32067 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
@@ -43,6 +43,11 @@ public abstract class CharacterType extends PrimitiveType {
return len;
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
// BE deserializes STRING into VARBINARY by copying the original
bytes, including embedded
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/DateLikeType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/DateLikeType.java
index f931ff2ce27..e388a15695f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/DateLikeType.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/DateLikeType.java
@@ -37,6 +37,11 @@ import java.time.LocalDateTime;
*/
public abstract class DateLikeType extends PrimitiveType implements
RangeScalable {
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
protected LocalDate toLocalDate(double d) {
// d = (year * 10000 + month * 100 + day) * 1000000L;
int date = (int) (d / 1000000);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/IntegralType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/IntegralType.java
index 7b36fd79945..483e925e14d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/IntegralType.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/IntegralType.java
@@ -50,6 +50,11 @@ public class IntegralType extends NumericType {
return "integral";
}
+ @Override
+ public boolean supportsValuePreservingGrouping() {
+ return true;
+ }
+
@Override
public boolean isInjectiveCastTo(DataType target) {
if (target instanceof IntegralType) {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDeltaTestBase.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDeltaTestBase.java
index 7ac4e359b7a..82255509d48 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDeltaTestBase.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDeltaTestBase.java
@@ -139,7 +139,11 @@ abstract class IvmDeltaTestBase {
/** Builds a scan for the given table id and name (for delta plan
generator tests). */
protected LogicalOlapScan buildScanForTable(long tableId, String
tableName) {
- OlapTable table = PlanConstructor.newOlapTable(tableId, tableName, 0);
+ return buildScanForTable(tableId, tableName, KeysType.DUP_KEYS);
+ }
+
+ protected LogicalOlapScan buildScanForTable(long tableId, String
tableName, KeysType keysType) {
+ OlapTable table = PlanConstructor.newOlapTable(tableId, tableName, 0,
keysType);
addTestPartition(table);
enableChangelog(table);
table.setQualifiedDbName("test_db");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandlerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandlerTest.java
index bfaf8af63f5..1ca7ac7a838 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandlerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandlerTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.mtmv.ivm;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.stream.OlapTableStream;
import org.apache.doris.common.Pair;
@@ -28,19 +29,30 @@ import
org.apache.doris.nereids.rules.exploration.join.JoinReorderContext;
import org.apache.doris.nereids.rules.expression.rules.FoldConstantRuleOnFE;
import org.apache.doris.nereids.trees.expressions.Add;
import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.IsNull;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Not;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.ConnectionId;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Length;
import org.apache.doris.nereids.trees.expressions.functions.scalar.Random;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Uuid;
+import org.apache.doris.nereids.trees.expressions.literal.BigIntLiteral;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
import
org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapTableStreamScan;
@@ -49,7 +61,10 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalSubQueryAlias;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
import org.apache.doris.nereids.types.BigIntType;
import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DoubleType;
import org.apache.doris.nereids.types.LargeIntType;
+import org.apache.doris.nereids.types.TinyIntType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -423,10 +438,230 @@ class IvmJoinDeltaHandlerTest extends IvmDeltaTestBase {
assertNoDuplicateScanRelationIds(result.plan);
}
+ @Test
+ void testUniqueNullSideDeltaUsesNetRepairForLeftOuterJoin() {
+ LogicalOlapScan leftSnapshot = buildSnapshotScanForTable(1201,
"unique_left_retained");
+ LogicalOlapScan rightDelta = buildUniqueScan(1202,
"unique_left_delta", false);
+ NormalizedOuterJoinPlan bundle =
normalizedOuterJoin(rowIdProject(leftSnapshot), rowIdProject(rightDelta));
+
+ assertUniqueNullSideEvents(bundle, 1);
+ }
+
+ @Test
+ void testUniqueNullSideDeltaUsesNetRepairForRightOuterJoin() {
+ LogicalOlapScan leftDelta = buildUniqueScan(1211,
"unique_right_delta", false);
+ LogicalOlapScan rightSnapshot = buildSnapshotScanForTable(1212,
"unique_right_retained");
+ NormalizedOuterJoinPlan bundle =
normalizedRightOuterJoin(rowIdProject(leftDelta),
+ rowIdProject(rightSnapshot));
+
+ assertUniqueNullSideEvents(bundle, 1);
+ }
+
+ @Test
+ void testCompositeUniqueNullSideKeyUsesNetRepair() {
+ Plan left = rowIdProject(buildSnapshotScanForTable(1221,
"composite_retained"));
+ Plan right = rowIdProject(buildUniqueScan(1222, "composite_delta",
true));
+ LogicalJoin<?, ?> join = new LogicalJoin<>(JoinType.LEFT_OUTER_JOIN,
+ ImmutableList.of(new EqualTo(left.getOutput().get(1),
right.getOutput().get(1)),
+ new EqualTo(left.getOutput().get(2),
right.getOutput().get(2))),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> topProject = normalizedJoinProject(join);
+ NormalizedOuterJoinPlan bundle = new
NormalizedOuterJoinPlan(topProject, left.getOutput().size(),
+ deterministicRewriteResult(topProject));
+
+ assertUniqueNullSideEvents(bundle, 2);
+ }
+
+ @Test
+ void testPartialCompositeUniqueKeyKeepsSnapshotChecks() {
+ LogicalOlapScan leftSnapshot = buildSnapshotScanForTable(1231,
"partial_key_retained");
+ LogicalOlapScan rightDelta = buildUniqueScan(1232,
"partial_key_delta", true);
+ NormalizedOuterJoinPlan bundle =
normalizedOuterJoin(rowIdProject(leftSnapshot), rowIdProject(rightDelta));
+
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(bundle.topProject,
+ newRefreshContext(bundle.topProject, bundle.rewriteResult));
+
+ Assertions.assertEquals(3,
findOnlyUnion(result.plan).children().size());
+ Assertions.assertEquals(2, result.plan.collectToList(node -> node
instanceof LogicalJoin
+ && ((LogicalJoin<?, ?>)
node).getJoinType().isLeftAntiJoin()).size());
+ }
+
+ @Test
+ void testUniqueNullSideExpressionKeyKeepsSnapshotChecks() {
+ Plan left = rowIdProject(buildSnapshotScanForTable(1241,
"expression_key_retained"));
+ Plan right = rowIdProject(buildUniqueScan(1242,
"expression_key_delta", false));
+ LogicalJoin<?, ?> join = new LogicalJoin<>(JoinType.LEFT_OUTER_JOIN,
+ ImmutableList.of(new EqualTo(firstUserSlot(left),
+ new Add(firstUserSlot(right), new
TinyIntLiteral((byte) 1)))),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> topProject = normalizedJoinProject(join);
+ IvmRewriteResult rewriteResult =
deterministicRewriteResult(topProject);
+
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(topProject,
+ newRefreshContext(topProject, rewriteResult));
+
+ Assertions.assertEquals(3,
findOnlyUnion(result.plan).children().size());
+ Assertions.assertEquals(2, result.plan.collectToList(node -> node
instanceof LogicalJoin
+ && ((LogicalJoin<?, ?>)
node).getJoinType().isLeftAntiJoin()).size());
+ }
+
+ @Test
+ void testUniqueNullSideResidualConditionCountsMatchesPerPredicateTuple() {
+ Plan left = rowIdProject(buildSnapshotScanForTable(1251,
"unique_residual_retained"));
+ Plan right = rowIdProject(buildUniqueScan(1252,
"unique_residual_delta", false));
+ LogicalJoin<?, ?> join = new LogicalJoin<>(JoinType.LEFT_OUTER_JOIN,
+ ImmutableList.of(new EqualTo(firstUserSlot(left),
firstUserSlot(right))),
+ ImmutableList.of(new GreaterThan(left.getOutput().get(2),
right.getOutput().get(2))),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> top = normalizedJoinProject(join);
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(top,
+ newRefreshContext(top, deterministicRewriteResult(top)));
+
+ LogicalUnion union = findOnlyUnion(result.plan);
+ Assertions.assertEquals(2, union.children().size());
+ LogicalProject<?> repair = (LogicalProject<?>) union.child(1);
+ LogicalJoin<?, ?> lookup = (LogicalJoin<?, ?>) repair.child();
+ LogicalAggregate<?> counts = (LogicalAggregate<?>)
lookup.right().child(0);
+
Assertions.assertEquals(ImmutableList.of(firstUserSlot(left).getName(),
left.getOutput().get(2).getName()),
+ counts.getGroupByExpressions().stream().map(expr -> ((Slot)
expr).getName())
+ .collect(ImmutableList.toImmutableList()));
+ Assertions.assertEquals(2, lookup.getHashJoinConjuncts().size());
+
Assertions.assertTrue(lookup.getHashJoinConjuncts().stream().allMatch(NullSafeEqual.class::isInstance));
+ LogicalJoin<?, ?> matches =
Assertions.assertInstanceOf(LogicalJoin.class, counts.child());
+ Assertions.assertEquals(1, matches.getOtherJoinConjuncts().size());
+ Slot netCount = counts.getOutput().get(2);
+ Expression factor = ((Alias)
repair.getProjects().get(repair.getProjects().size() - 2)).child();
+ // Three retained rows sharing the tuple require three unit repairs,
not three repairs of magnitude three.
+ Assertions.assertEquals(BigInteger.ONE.negate(),
evaluateNetCount(factor, netCount, 3));
+ Assertions.assertEquals(BigInteger.ONE, evaluateNetCount(factor,
netCount, -3));
+ assertNoDuplicateScanRelationIds(result.plan);
+ }
+
+ @Test
+ void testFloatingPointResidualInputKeepsSnapshotRepair() {
+ Plan scan = rowIdProject(buildSnapshotScanForTable(1291,
"float_retained"));
+ Alias score = new Alias(new Cast(firstUserSlot(scan),
DoubleType.INSTANCE), "score");
+ Plan left = new
LogicalProject<>(ImmutableList.<NamedExpression>builder()
+ .addAll(scan.getOutput()).add(score).build(), scan);
+ Plan right = rowIdProject(buildUniqueScan(1292, "float_unique_delta",
false));
+ LogicalJoin<?, ?> join = new LogicalJoin<>(JoinType.LEFT_OUTER_JOIN,
+ ImmutableList.of(new EqualTo(firstUserSlot(left),
firstUserSlot(right))),
+ ImmutableList.of(new GreaterThan(score.toSlot(),
firstUserSlot(right))),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> top = normalizedJoinProject(join);
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(top,
+ newRefreshContext(top, deterministicRewriteResult(top)));
+ Assertions.assertEquals(3, findOnlyUnion(result.plan).arity());
+ }
+
+ @Test
+ void testResidualOnVolatileRetainedAliasKeepsSnapshotRepair() {
+ Plan scan = rowIdProject(buildSnapshotScanForTable(1301,
"volatile_retained"));
+ Alias tag = new Alias(new Uuid(), "tag");
+ Plan left = new
LogicalProject<>(ImmutableList.<NamedExpression>builder()
+ .addAll(scan.getOutput()).add(tag).build(), scan);
+ Plan right = rowIdProject(buildUniqueScan(1302,
"volatile_unique_delta", false));
+ LogicalJoin<?, ?> join = new LogicalJoin<>(JoinType.LEFT_OUTER_JOIN,
+ ImmutableList.of(new EqualTo(firstUserSlot(left),
firstUserSlot(right))),
+ ImmutableList.of(new GreaterThan(new Length(tag.toSlot()),
firstUserSlot(right))),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> top = normalizedJoinProject(join);
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(top,
+ newRefreshContext(top, deterministicRewriteResult(top)));
+ Assertions.assertEquals(3, findOnlyUnion(result.plan).arity());
+ Assertions.assertEquals(2, result.plan.collectToList(node -> node
instanceof LogicalJoin
+ && ((LogicalJoin<?, ?>)
node).getJoinType().isLeftAntiJoin()).size());
+ }
+
+ @Test
+ void testSessionDependentResidualKeepsSnapshotRepair() {
+ Plan left = rowIdProject(buildSnapshotScanForTable(1281,
"session_retained"));
+ Plan right = rowIdProject(buildUniqueScan(1282,
"session_unique_delta", false));
+ LogicalJoin<?, ?> join = new LogicalJoin<>(JoinType.LEFT_OUTER_JOIN,
+ ImmutableList.of(new EqualTo(firstUserSlot(left),
firstUserSlot(right))),
+ ImmutableList.of(new GreaterThan(firstUserSlot(left), new
ConnectionId())),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> top = normalizedJoinProject(join);
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(top,
+ newRefreshContext(top, deterministicRewriteResult(top)));
+ Assertions.assertEquals(3, findOnlyUnion(result.plan).arity());
+ Assertions.assertEquals(2, result.plan.collectToList(node -> node
instanceof LogicalJoin
+ && ((LogicalJoin<?, ?>)
node).getJoinType().isLeftAntiJoin()).size());
+ }
+
+ private LogicalOlapScan buildUniqueScan(long tableId, String name, boolean
compositeKey) {
+ LogicalOlapScan scan = buildScanForTable(tableId, name,
KeysType.UNIQUE_KEYS);
+ if (!compositeKey) {
+ scan.getTable().getColumn("name").setIsKey(false);
+ }
+ return scan;
+ }
+
+ private void assertUniqueNullSideEvents(NormalizedOuterJoinPlan bundle,
int eventKeyCount) {
+ for (DataType sequenceType : ImmutableList.of(BigIntType.INSTANCE,
LargeIntType.INSTANCE)) {
+ assertUniqueNullSideEvents(bundle, eventKeyCount, sequenceType);
+ }
+ }
+
+ private void assertUniqueNullSideEvents(NormalizedOuterJoinPlan bundle,
int eventKeyCount, DataType sequenceType) {
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler(sequenceType)
+ .exposeRewritePlan(bundle.topProject,
newRefreshContext(bundle.topProject, bundle.rewriteResult));
+ LogicalUnion union = findOnlyUnion(result.plan);
+ Assertions.assertEquals(2, union.children().size());
+ assertUnionChildrenAlign(union);
+ assertUnionOutputsDoNotReuseChildExprIds(union);
+ assertNoDuplicateScanRelationIds(result.plan);
+ Assertions.assertFalse(bundle.rewriteResult.isAggMv(),
+ "An internal repair aggregate must not turn the join MV into
an aggregate MV");
+ Assertions.assertTrue(result.plan.collectToList(node -> node
instanceof LogicalJoin
+ && ((LogicalJoin<?, ?>)
node).getJoinType().isLeftAntiJoin()).isEmpty());
+ List<LogicalAggregate> aggregates = union.collectToList(node -> node
instanceof LogicalAggregate);
+ Assertions.assertEquals(1, aggregates.size());
+ LogicalAggregate<?> aggregate = aggregates.get(0);
+ Assertions.assertEquals(eventKeyCount,
aggregate.getGroupByExpressions().size());
+ LogicalFilter<?> nonNullKeys =
Assertions.assertInstanceOf(LogicalFilter.class, aggregate.child());
+ for (Expression key : aggregate.getGroupByExpressions()) {
+ Assertions.assertTrue(nonNullKeys.getConjuncts().contains(new
Not(new IsNull(key))),
+ "A NULL join key must not generate a repair event");
+ }
+ Assertions.assertEquals(1, aggregate.getOutputExpressions().stream()
+ .filter(expr -> expr.anyMatch(child -> child instanceof
Sum)).count());
+ Assertions.assertTrue(union.collectToList(node -> node instanceof
LogicalOlapScan).stream()
+ .allMatch(scan ->
IvmDeltaRewriteHelper.INSTANCE.isIncrementalDeltaScan((LogicalOlapScan) scan)),
+ "Unique-key event generation must not read null-side
snapshots");
+
+ LogicalProject<?> repair = (LogicalProject<?>) union.child(1);
+ for (int i = eventKeyCount; i < repair.getProjects().size() - 2; i++) {
+ Assertions.assertInstanceOf(NullLiteral.class, ((Alias)
repair.getProjects().get(i)).child());
+ }
+ NamedExpression factor =
repair.getProjects().get(repair.getProjects().size() - 2);
+ Assertions.assertEquals(Column.IVM_DML_FACTOR_COL, factor.getName());
+ Assertions.assertEquals(TinyIntType.INSTANCE, factor.getDataType());
+ Alias sequence = (Alias)
repair.getProjects().get(repair.getProjects().size() - 1);
+ Assertions.assertEquals(Column.SEQUENCE_COL, sequence.getName());
+ Assertions.assertEquals(sequenceType, sequence.getDataType());
+ Slot netCount = aggregate.getOutput().get(eventKeyCount);
+ // Unlike the generic repair branches, this factor is computed from a
slot. The sign of -netCount
+ // must still order NULL-row reinsertion after retraction for both
sequence encodings.
+ BigInteger deleteSequence = evaluateNetCount(sequence.child(),
netCount, 1);
+ BigInteger insertSequence = evaluateNetCount(sequence.child(),
netCount, -1);
+ Assertions.assertFalse(deleteSequence.testBit(0));
+ Assertions.assertTrue(insertSequence.testBit(0));
+ Assertions.assertTrue(insertSequence.compareTo(deleteSequence) > 0);
+ Assertions.assertEquals(deleteSequence.shiftRight(1),
insertSequence.shiftRight(1));
+ }
+
+ private BigInteger evaluateNetCount(Expression expression, Slot netCount,
long value) {
+ Expression bound = ExpressionUtils.replace(expression,
ImmutableMap.of(netCount, new BigIntLiteral(value)));
+ Expression folded = FoldConstantRuleOnFE.evaluateWithoutContext(bound);
+ Assertions.assertInstanceOf(Literal.class, folded);
+ return new BigInteger(((Literal) folded).getStringValue());
+ }
+
@Test
void
testFullOuterJoinLeftDeltaUsesLeftOuterJoinAndRepairsRightDanglingRows() {
- LogicalOlapScan leftDelta = buildScanForTable(1, "t1");
- LogicalOlapScan rightSnapshot = buildSnapshotScanForTable(2, "t2");
+ LogicalOlapScan leftDelta = buildScanForTable(1271, "full_left_delta");
+ LogicalOlapScan rightSnapshot = buildSnapshotScanForTable(1272,
"full_right_snapshot");
NormalizedOuterJoinPlan bundle =
normalizedFullOuterJoin(rowIdProject(leftDelta), rowIdProject(rightSnapshot));
TestableIvmJoinDeltaHandler handler = new
TestableIvmJoinDeltaHandler();
@@ -437,11 +672,7 @@ class IvmJoinDeltaHandlerTest extends IvmDeltaTestBase {
LogicalProject<?> joinOutputProject = (LogicalProject<?>)
((LogicalProject<?>) result.plan).child();
Assertions.assertInstanceOf(LogicalUnion.class,
joinOutputProject.child());
LogicalUnion union = (LogicalUnion) joinOutputProject.child();
- Assertions.assertEquals(3, union.children().size());
- Assertions.assertTrue(union.child(0).collectToList(node -> node
instanceof LogicalJoin
- && ((LogicalJoin<?, ?>)
node).getJoinType().isOuterJoin()).size() >= 1);
- assertLeftNullSideRowId((LogicalProject<?>) union.child(1),
bundle.leftOutputSize, (byte) -1);
- assertLeftNullSideRowId((LogicalProject<?>) union.child(2),
bundle.leftOutputSize, (byte) 1);
+ assertFullOuterDetailAndRepair(union, JoinType.LEFT_OUTER_JOIN, 2);
assertUnionChildrenAlign(union);
assertNoDuplicateScanRelationIds(result.plan);
}
@@ -460,14 +691,35 @@ class IvmJoinDeltaHandlerTest extends IvmDeltaTestBase {
LogicalProject<?> joinOutputProject = (LogicalProject<?>)
((LogicalProject<?>) result.plan).child();
Assertions.assertInstanceOf(LogicalUnion.class,
joinOutputProject.child());
LogicalUnion union = (LogicalUnion) joinOutputProject.child();
- Assertions.assertEquals(3, union.children().size());
- assertProjectedJoinBranch(union.child(0), JoinType.RIGHT_OUTER_JOIN);
- assertRightNullSideRowId((LogicalProject<?>) union.child(1),
bundle.leftOutputSize, (byte) -1);
- assertRightNullSideRowId((LogicalProject<?>) union.child(2),
bundle.leftOutputSize, (byte) 1);
+ assertFullOuterDetailAndRepair(union, JoinType.RIGHT_OUTER_JOIN, 2);
assertUnionChildrenAlign(union);
assertNoDuplicateScanRelationIds(result.plan);
}
+ @Test
+ void testFullOuterUniqueDeltaUsesOneNetRepair() {
+ for (boolean leftDelta : ImmutableList.of(true, false)) {
+ Plan delta = rowIdProject(buildUniqueScan(1262,
"full_unique_delta", false));
+ Plan snapshot = rowIdProject(buildSnapshotScanForTable(1261,
"full_retained"));
+ NormalizedOuterJoinPlan bundle = leftDelta ?
normalizedFullOuterJoin(delta, snapshot)
+ : normalizedFullOuterJoin(snapshot, delta);
+ IvmDeltaRewriteResult result = new
TestableIvmJoinDeltaHandler().exposeRewritePlan(bundle.topProject,
+ newRefreshContext(bundle.topProject,
bundle.rewriteResult));
+ LogicalUnion union = findOnlyUnion(result.plan);
+ assertFullOuterDetailAndRepair(union, leftDelta ?
JoinType.LEFT_OUTER_JOIN : JoinType.RIGHT_OUTER_JOIN, 0);
+ Assertions.assertEquals(1, union.child(1).collectToList(node ->
node instanceof LogicalAggregate).size());
+ assertNoDuplicateScanRelationIds(result.plan);
+ }
+ }
+
+ private void assertFullOuterDetailAndRepair(LogicalUnion union, JoinType
detailType, int expectedAntiJoins) {
+ Assertions.assertEquals(2, union.children().size());
+ assertProjectedJoinBranch(union.child(0), detailType);
+ assertProjectedJoinBranch(union.child(1), JoinType.INNER_JOIN);
+ Assertions.assertEquals(expectedAntiJoins,
union.child(1).collectToList(node -> node instanceof LogicalJoin
+ && ((LogicalJoin<?, ?>)
node).getJoinType().isLeftAntiJoin()).size());
+ }
+
@Test
void testNullSideDeltaExtractsHashConjunctFromOtherConjuncts() {
LogicalOlapScan leftSnapshot = buildSnapshotScanForTable(1, "t1");
diff --git
a/regression-test/data/mtmv_p0/ivm/test_ivm_unique_key_null_repair.out
b/regression-test/data/mtmv_p0/ivm/test_ivm_unique_key_null_repair.out
new file mode 100644
index 00000000000..7ba4a50d0dc
--- /dev/null
+++ b/regression-test/data/mtmv_p0/ivm/test_ivm_unique_key_null_repair.out
@@ -0,0 +1,219 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !order_baseline_values --
+\N 1 20 null_row_id \N \N
+1 1 10 first 20 2
+2 1 20 second \N \N
+3 2 10 other_org 30 1
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key 20 2
+6 1 30 unmatched \N \N
+8 1 20 same_on_inputs \N \N
+
+-- !order_baseline_difference_0 --
+
+-- !order_baseline_difference_1 --
+
+-- !order_baseline_difference_2 --
+
+-- !order_baseline_difference_3 --
+
+-- !order_baseline_difference_4 --
+
+-- !order_baseline_difference_5 --
+
+-- !order_baseline_difference_6 --
+
+-- !order_first_match_values --
+\N 1 20 null_row_id 5 1
+1 1 10 first 20 2
+2 1 20 second 5 1
+3 2 10 other_org 30 1
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key 20 2
+6 1 30 unmatched \N \N
+8 1 20 same_on_inputs 5 1
+
+-- !order_first_match_difference_0 --
+
+-- !order_first_match_difference_1 --
+
+-- !order_first_match_difference_2 --
+
+-- !order_first_match_difference_3 --
+
+-- !order_first_match_difference_4 --
+
+-- !order_first_match_difference_5 --
+
+-- !order_first_match_difference_6 --
+
+-- !order_same_key_updates_values --
+\N 1 20 null_row_id 5 1
+1 1 10 first 45 2
+2 1 20 second 5 1
+3 2 10 other_org 30 1
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key 45 2
+6 1 30 unmatched \N \N
+8 1 20 same_on_inputs 5 1
+
+-- !order_same_key_updates_difference_0 --
+
+-- !order_same_key_updates_difference_1 --
+
+-- !order_same_key_updates_difference_2 --
+
+-- !order_same_key_updates_difference_3 --
+
+-- !order_same_key_updates_difference_4 --
+
+-- !order_same_key_updates_difference_5 --
+
+-- !order_same_key_updates_difference_6 --
+
+-- !order_residual_loses_match_values --
+\N 1 20 null_row_id 5 1
+1 1 10 first 10 2
+2 1 20 second 5 1
+3 2 10 other_org 30 1
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key 10 2
+6 1 30 unmatched \N \N
+8 1 20 same_on_inputs 5 1
+
+-- !order_residual_loses_match_difference_0 --
+
+-- !order_residual_loses_match_difference_1 --
+
+-- !order_residual_loses_match_difference_2 --
+
+-- !order_residual_loses_match_difference_3 --
+
+-- !order_residual_loses_match_difference_4 --
+
+-- !order_residual_loses_match_difference_5 --
+
+-- !order_residual_loses_match_difference_6 --
+
+-- !order_last_match_values --
+\N 1 20 null_row_id \N \N
+1 1 10 first 10 2
+2 1 20 second \N \N
+3 2 10 other_org 35 2
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key 10 2
+6 1 30 unmatched \N \N
+8 1 20 same_on_inputs \N \N
+
+-- !order_last_match_difference_0 --
+
+-- !order_last_match_difference_1 --
+
+-- !order_last_match_difference_2 --
+
+-- !order_last_match_difference_3 --
+
+-- !order_last_match_difference_4 --
+
+-- !order_last_match_difference_5 --
+
+-- !order_last_match_difference_6 --
+
+-- !order_null_payload_match_values --
+\N 1 20 null_row_id \N 1
+1 1 10 first 10 2
+2 1 20 second \N 1
+3 2 10 other_org 35 2
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key 10 2
+6 1 30 unmatched \N \N
+8 1 20 same_on_inputs \N 1
+
+-- !order_null_payload_match_difference_0 --
+
+-- !order_null_payload_match_difference_1 --
+
+-- !order_null_payload_match_difference_2 --
+
+-- !order_null_payload_match_difference_3 --
+
+-- !order_null_payload_match_difference_4 --
+
+-- !order_null_payload_match_difference_5 --
+
+-- !order_null_payload_match_difference_6 --
+
+-- !order_key_migration_and_both_sides_values --
+\N 1 20 null_row_id \N 1
+1 1 10 first \N \N
+2 1 20 second \N 1
+3 2 10 other_org 35 2
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key \N \N
+6 1 30 changed 10 2
+7 1 30 new_contact 10 2
+8 1 20 same_on_inputs \N 1
+
+-- !order_key_migration_and_both_sides_difference_0 --
+
+-- !order_key_migration_and_both_sides_difference_1 --
+
+-- !order_key_migration_and_both_sides_difference_2 --
+
+-- !order_key_migration_and_both_sides_difference_3 --
+
+-- !order_key_migration_and_both_sides_difference_4 --
+
+-- !order_key_migration_and_both_sides_difference_5 --
+
+-- !order_key_migration_and_both_sides_difference_6 --
+
+-- !order_batch_cancellation_values --
+\N 1 20 null_row_id \N 1
+1 1 10 first \N \N
+2 1 20 second \N 1
+3 2 10 other_org 37 2
+4 1 \N null_key \N \N
+5 1 10 duplicate_join_key \N \N
+6 1 30 changed 10 2
+7 1 30 new_contact 10 2
+8 1 20 same_on_inputs \N 1
+
+-- !order_batch_cancellation_difference_0 --
+
+-- !order_batch_cancellation_difference_1 --
+
+-- !order_batch_cancellation_difference_2 --
+
+-- !order_batch_cancellation_difference_3 --
+
+-- !order_batch_cancellation_difference_4 --
+
+-- !order_batch_cancellation_difference_5 --
+
+-- !order_batch_cancellation_difference_6 --
+
+-- !order_sign_flip_ivm_signed_zero_unique_difference --
+
+-- !order_sign_flip_ivm_signed_zero_right_difference --
+
+-- !order_sign_flip_ivm_signed_zero_full_unique_difference --
+
+-- !order_sign_flip_ivm_signed_zero_nonunique_difference --
+
+-- !order_sign_flip_and_insert_ivm_signed_zero_unique_difference --
+
+-- !order_sign_flip_and_insert_ivm_signed_zero_right_difference --
+
+-- !order_sign_flip_and_insert_ivm_signed_zero_full_unique_difference --
+
+-- !order_sign_flip_and_insert_ivm_signed_zero_nonunique_difference --
+
+-- !order_last_match_and_both_sides_ivm_signed_zero_unique_difference --
+
+-- !order_last_match_and_both_sides_ivm_signed_zero_right_difference --
+
+-- !order_last_match_and_both_sides_ivm_signed_zero_full_unique_difference --
+
+-- !order_last_match_and_both_sides_ivm_signed_zero_nonunique_difference --
+
diff --git a/regression-test/suites/mtmv_p0/ivm/test_ivm_explain_refresh.groovy
b/regression-test/suites/mtmv_p0/ivm/test_ivm_explain_refresh.groovy
index 3a272189c79..2f6f3fb8fa3 100644
--- a/regression-test/suites/mtmv_p0/ivm/test_ivm_explain_refresh.groovy
+++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_explain_refresh.groovy
@@ -150,6 +150,10 @@ suite("test_ivm_explain_refresh") {
'num_nulls'='0', 'min_value'='100', 'max_value'='300');
"""
+ // Only t1 has pending changes. WITH ALL STREAMS includes the exhausted t2
stream
+ // during delta rewriting, but t2's unique join key allows snapshot-free
NULL repair.
+ // Its empty detail/aggregate contribution is removed during optimization,
so the
+ // final shape does not retain the old UNION and two snapshot ANTI JOIN
branches.
explainIvmPlanWithoutStreamId("all_streams_shape_plan", """
EXPLAIN SHAPE PLAN REFRESH MATERIALIZED VIEW
test_ivm_explain_refresh_mv INCREMENTAL WITH ALL STREAMS
""")
@@ -175,7 +179,13 @@ suite("test_ivm_explain_refresh") {
contains "LogicalJoin"
contains "LEFT_OUTER_JOIN"
contains "test_ivm_explain_refresh_t1"
- contains "test_ivm_explain_refresh_t2"
+ // t2 is read through its incremental/snapshot stream scans.
Unique-key repair
+ // no longer adds the ordinary t2 post-snapshot scan that exposed its
base name.
+ contains "readMode=INCREMENTAL"
+ contains "readMode=SNAPSHOT"
+ contains "LogicalAggregate"
+ contains "__DORIS_IVM_NULL_SIDE_NET_COUNT__"
+ notContains "LEFT_ANTI_JOIN"
}
explain {
diff --git
a/regression-test/suites/mtmv_p0/ivm/test_ivm_unique_key_null_repair.groovy
b/regression-test/suites/mtmv_p0/ivm/test_ivm_unique_key_null_repair.groovy
new file mode 100644
index 00000000000..629ad6c344a
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/ivm/test_ivm_unique_key_null_repair.groovy
@@ -0,0 +1,303 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.awaitility.Awaitility
+
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_ivm_unique_key_null_repair") {
+ sql "SET enable_materialized_view_rewrite = false"
+ sql "SET enable_dml_materialized_view_rewrite = false"
+
+ def downstream = ["ivm_unique_repair_left", "ivm_unique_repair_right",
+ "ivm_unique_repair_partial_key",
"ivm_unique_repair_residual",
+ "ivm_unique_repair_full",
"ivm_unique_repair_full_residual",
+ "ivm_unique_repair_right_residual"]
+ downstream.each { sql "DROP MATERIALIZED VIEW IF EXISTS ${it}" }
+ sql "DROP MATERIALIZED VIEW IF EXISTS ivm_unique_repair_groups"
+ sql "DROP TABLE IF EXISTS ivm_unique_repair_contacts"
+ sql "DROP TABLE IF EXISTS ivm_unique_repair_details"
+
+ sql """
+ CREATE TABLE ivm_unique_repair_contacts (
+ id INT NULL, organisationid INT, contactid INT,
+ label VARCHAR(32), threshold_value INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1", "enable_unique_key_merge_on_write" =
"true",
+ "changelog.enable" = "true", "changelog.need_historical_value" =
"true"
+ )
+ """
+ sql """
+ CREATE TABLE ivm_unique_repair_details (
+ id INT NOT NULL, organisationid INT, contactid INT, amount INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1", "enable_unique_key_merge_on_write" =
"true",
+ "changelog.enable" = "true", "changelog.need_historical_value" =
"true"
+ )
+ """
+ sql """INSERT INTO ivm_unique_repair_contacts VALUES
+ (1, 1, 10, 'first', 15), (2, 1, 20, 'second', 15),
+ (3, 2, 10, 'other_org', 15), (4, 1, NULL, 'null_key', 15),
+ (5, 1, 10, 'duplicate_join_key', 25), (6, 1, 30, 'unmatched', 15),
+ (NULL, 1, 20, 'null_row_id', NULL), (8, 1, 20, 'same_on_inputs', NULL)
+ """
+ sql """INSERT INTO ivm_unique_repair_details VALUES
+ (1, 1, 10, 10), (2, 1, 10, 10), (3, 2, 10, 30), (4, 1, NULL, 40)
+ """
+
+ // GROUP BY proves the complete business key unique; the L2 stream can
still contain
+ // several versions of that key. A SUM payload update must retain
-old/+new detail events.
+ sql """
+ CREATE MATERIALIZED VIEW ivm_unique_repair_groups
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1", "changelog.enable" = "true",
+ "changelog.need_historical_value" = "true"
+ )
+ AS SELECT organisationid, contactid, SUM(amount) AS total, COUNT(*) AS
cnt
+ FROM ivm_unique_repair_details GROUP BY organisationid, contactid
+ """
+
+ // CreateTime has second granularity and TaskId is not monotonically
increasing.
+ // Capture existing ids, then wait for the exact task added by this
refresh.
+ def taskIdsOf = { String mv ->
+ sql_return_maparray("""
+ SELECT TaskId FROM tasks('type'='mv')
+ WHERE MvDatabaseName = '${context.dbName}' AND MvName = '${mv}'
+ """).collect { it.TaskId.toString() } as Set
+ }
+ def refresh = { String mv, String mode ->
+ def previousIds = taskIdsOf(mv)
+ sql "REFRESH MATERIALIZED VIEW ${mv} ${mode}"
+ String taskId = null
+ Awaitility.await().atMost(300, SECONDS).pollInterval(1, SECONDS).until
{
+ taskId = taskIdsOf(mv).find { !previousIds.contains(it) }
+ return taskId != null
+ }
+ def latest
+ Awaitility.await().atMost(300, SECONDS).pollInterval(1, SECONDS).until
{
+ latest = sql_return_maparray("""
+ SELECT Status, RefreshMode, ErrorMsg FROM tasks('type'='mv')
+ WHERE MvDatabaseName = '${context.dbName}' AND MvName = '${mv}'
+ AND TaskId = '${taskId}'
+ """)
+ return !latest.isEmpty() && !['PENDING',
'RUNNING'].contains(latest[0].Status.toString())
+ }
+ assertEquals('SUCCESS', latest[0].Status.toString())
+ if (mode == 'INCREMENTAL') {
+ assertTrue(latest[0].RefreshMode.toString() != 'COMPLETE',
+ "Unexpected complete fallback for ${mv}: ${latest}")
+ }
+ }
+ refresh('ivm_unique_repair_groups', 'COMPLETE')
+
+ def fullKey = 'a.organisationid = b.organisationid AND a.contactid =
b.contactid'
+ def residual = "${fullKey} AND b.total > COALESCE(a.threshold_value, 0)"
+ def conditions = [
+ (downstream[0]): fullKey,
+ (downstream[1]): fullKey,
+ // organisationid alone is not unique: a net positive change can be
1->2, not 0->1.
+ (downstream[2]): 'a.organisationid = b.organisationid',
+ // A value update may change matching despite an unchanged number of
L1 rows.
+ (downstream[3]): residual,
+ (downstream[4]): fullKey,
+ (downstream[5]): residual,
+ (downstream[6]): residual
+ ]
+ def fromClause = { mv, groups ->
+ if (mv in [downstream[1], downstream[6]]) {
+ return "${groups} b RIGHT JOIN ivm_unique_repair_contacts a"
+ }
+ def joinType = mv in [downstream[4], downstream[5]] ? 'FULL OUTER
JOIN' : 'LEFT JOIN'
+ return "ivm_unique_repair_contacts a ${joinType} ${groups} b"
+ }
+ downstream.each { mv ->
+ sql """
+ CREATE MATERIALIZED VIEW ${mv}
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ AS SELECT a.id, a.organisationid, a.contactid, a.label, b.total,
b.cnt
+ FROM ${fromClause(mv, "ivm_unique_repair_groups")} ON
${conditions[mv]}
+ """
+ refresh(mv, 'COMPLETE')
+ }
+
+ def columns = 'id, organisationid, contactid, label, total, cnt'
+ def fullGroups = """(SELECT organisationid, contactid, SUM(amount) AS
total, COUNT(*) AS cnt
+ FROM ivm_unique_repair_details GROUP BY organisationid, contactid)"""
+ def check = { String stage ->
+ delegate.quickRunTest("order_${stage}_values",
+ "SELECT ${columns} FROM ivm_unique_repair_left", true)
+ downstream.eachWithIndex { mv, index ->
+ // Compare signed bag multiplicities, including NULLs and
duplicate join matches,
+ // against a fresh query over the original tables instead of the
upstream MV.
+ delegate.quickRunTest("order_${stage}_difference_${index}", """
+ SELECT ${columns}, SUM(weight) AS multiplicity_difference
+ FROM (
+ SELECT ${columns}, 1 AS weight FROM ${mv}
+ UNION ALL
+ SELECT a.id, a.organisationid, a.contactid, a.label,
b.total, b.cnt, -1 AS weight
+ FROM ${fromClause(mv, fullGroups)}
+ ON ${conditions[mv]}
+ ) differences
+ GROUP BY ${columns}
+ HAVING SUM(weight) <> 0
+ """, true)
+ }
+ }
+ def refreshAndCheck = { String stage ->
+ refresh('ivm_unique_repair_groups', 'INCREMENTAL')
+ if (stage == 'first_match') {
+ // Include both streams so the plan check does not depend on
pruning the idle stream.
+ // The grouped side uses signed aggregate repair in all three
joins. FULL JOIN also
+ // repairs from the contacts side, whose join key is not unique,
using two anti joins.
+ [downstream[3], downstream[5], downstream[6]].each { mv ->
+ explain {
+ sql "ANALYZED PLAN REFRESH MATERIALIZED VIEW ${mv}
INCREMENTAL WITH ALL STREAMS"
+ contains "__DORIS_IVM_NULL_SIDE_NET_COUNT__"
+ multiContains "type=LEFT_ANTI_JOIN,", mv == downstream[5]
? 2 : 0
+ }
+ }
+ }
+ downstream.each { refresh(it, 'INCREMENTAL') }
+ check(stage)
+ }
+ check('baseline')
+
+ // First match for key (1,20): retract its NULL row; a second
organisation-1 group
+ // simultaneously exercises the nonunique-key fallback.
+ sql "INSERT INTO ivm_unique_repair_details VALUES (5, 1, 20, 5)"
+ refreshAndCheck('first_match')
+
+ // Multiple same-key payload versions have net row count zero. The
residual predicate
+ // changes from true to false, then back above the two contacts' different
thresholds.
+ sql "UPDATE ivm_unique_repair_details SET amount = 1 WHERE id = 1"
+ sql "UPDATE ivm_unique_repair_details SET amount = 35 WHERE id = 1"
+ refreshAndCheck('same_key_updates')
+ sql "UPDATE ivm_unique_repair_details SET amount = 0 WHERE id = 1"
+ refreshAndCheck('residual_loses_match')
+
+ // Last match disappears, while another group's payload changes and NULL
keys change.
+ sql "DELETE FROM ivm_unique_repair_details WHERE id = 5"
+ sql "INSERT INTO ivm_unique_repair_details VALUES (6, 2, 10, 5), (7, 1,
NULL, 7)"
+ refreshAndCheck('last_match')
+
+ // A real matching row with NULL payload must remain distinguishable from
a NULL extension.
+ sql "INSERT INTO ivm_unique_repair_details VALUES (9, 1, 20, NULL)"
+ refreshAndCheck('null_payload_match')
+
+ // Move a whole group between business keys and change the retained side
in the same batch.
+ sql "UPDATE ivm_unique_repair_details SET contactid = 30 WHERE
organisationid = 1 AND contactid = 10"
+ sql "UPDATE ivm_unique_repair_contacts SET label = 'changed',
threshold_value = 5 WHERE id = 6"
+ sql "INSERT INTO ivm_unique_repair_contacts VALUES (7, 1, 30,
'new_contact', 5)"
+ refreshAndCheck('key_migration_and_both_sides')
+
+ // Delete/reinsert an existing group and insert/delete a new group before
refresh.
+ sql "DELETE FROM ivm_unique_repair_details WHERE id = 3"
+ sql "INSERT INTO ivm_unique_repair_details VALUES (3, 2, 10, 32), (8, 1,
40, 8)"
+ sql "DELETE FROM ivm_unique_repair_details WHERE id = 8"
+ refreshAndCheck('batch_cancellation')
+
+ sql "DROP MATERIALIZED VIEW IF EXISTS ivm_signed_zero_unique"
+ sql "DROP MATERIALIZED VIEW IF EXISTS ivm_signed_zero_right"
+ sql "DROP MATERIALIZED VIEW IF EXISTS ivm_signed_zero_full_unique"
+ sql "DROP MATERIALIZED VIEW IF EXISTS ivm_signed_zero_nonunique"
+ sql "DROP TABLE IF EXISTS ivm_signed_zero_a"
+ sql "DROP TABLE IF EXISTS ivm_signed_zero_b"
+ sql """
+ CREATE TABLE ivm_signed_zero_a (
+ id INT NOT NULL, k INT NOT NULL, score DOUBLE NOT NULL, score_f
FLOAT NOT NULL
+ )
+ UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1", "enable_unique_key_merge_on_write" =
"true",
+ "changelog.enable" = "true", "changelog.need_historical_value" =
"true"
+ )
+ """
+ sql """
+ CREATE TABLE ivm_signed_zero_b (k INT NOT NULL, score DOUBLE NOT NULL,
score_f FLOAT NOT NULL, v INT)
+ UNIQUE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1", "enable_unique_key_merge_on_write" =
"true",
+ "changelog.enable" = "true", "changelog.need_historical_value" =
"true"
+ )
+ """
+ sql """INSERT INTO ivm_signed_zero_a VALUES
+ (1, 1, CAST('+0.0' AS DOUBLE), CAST('+0.0' AS FLOAT)),
+ (2, 1, CAST('-0.0' AS DOUBLE), CAST('-0.0' AS FLOAT))
+ """
+ sql "INSERT INTO ivm_signed_zero_b VALUES (1, CAST('+0.0' AS DOUBLE),
CAST('+0.0' AS FLOAT), 101)"
+ def signedZeroJoins = [
+ ['ivm_signed_zero_unique', 'ivm_signed_zero_a a LEFT JOIN
ivm_signed_zero_b b',
+ 'a.k = b.k AND a.score = b.score', 2],
+ ['ivm_signed_zero_right', 'ivm_signed_zero_b b RIGHT JOIN
ivm_signed_zero_a a',
+ 'a.k = b.k AND a.score_f = b.score_f', 2],
+ ['ivm_signed_zero_full_unique', 'ivm_signed_zero_a a FULL OUTER JOIN
ivm_signed_zero_b b',
+ 'a.k = b.k AND a.score = b.score', 4],
+ ['ivm_signed_zero_nonunique', 'ivm_signed_zero_a a FULL OUTER JOIN
ivm_signed_zero_b b',
+ 'a.score = b.score', 4]
+ ]
+ signedZeroJoins.each { mv, from, predicate, repairBranches ->
+ sql """
+ CREATE MATERIALIZED VIEW ${mv}
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL DISTRIBUTED BY RANDOM
BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ AS SELECT a.id, b.v FROM ${from} ON ${predicate}
+ """
+ refresh(mv, 'COMPLETE')
+ explain {
+ sql "ANALYZED PLAN REFRESH MATERIALIZED VIEW ${mv} INCREMENTAL
WITH ALL STREAMS"
+ multiContains "type=LEFT_SEMI_JOIN,", repairBranches
+ multiContains "type=LEFT_ANTI_JOIN,", repairBranches
+ }
+ }
+ def checkSignedZeroJoins = { String stage ->
+ signedZeroJoins.each { mv, from, predicate, repairBranches ->
+ refresh(mv, 'INCREMENTAL')
+ delegate.quickRunTest("order_${stage}_${mv}_difference", """
+ SELECT id, v, SUM(weight)
+ FROM (
+ SELECT id, v, 1 AS weight FROM ${mv}
+ UNION ALL
+ SELECT a.id, b.v, -1 AS weight FROM ${from} ON ${predicate}
+ ) differences
+ GROUP BY id, v
+ HAVING SUM(weight) <> 0
+ """, true)
+ }
+ }
+
+ // A hash join distinguishes signed zeros. The update switches which A row
needs NULL padding.
+ sql "INSERT INTO ivm_signed_zero_b VALUES (1, CAST('-0.0' AS DOUBLE),
CAST('-0.0' AS FLOAT), 202)"
+ checkSignedZeroJoins('sign_flip')
+ sql """INSERT INTO ivm_signed_zero_b VALUES
+ (1, CAST('+0.0' AS DOUBLE), CAST('+0.0' AS FLOAT), 303),
+ (2, CAST('-0.0' AS DOUBLE), CAST('-0.0' AS FLOAT), 404)
+ """
+ checkSignedZeroJoins('sign_flip_and_insert')
+ sql "INSERT INTO ivm_signed_zero_a VALUES (3, 2, CAST('-0.0' AS DOUBLE),
CAST('-0.0' AS FLOAT))"
+ sql "DELETE FROM ivm_signed_zero_b WHERE k = 1"
+ checkSignedZeroJoins('last_match_and_both_sides')
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]