This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dd456ab7a89 [improvement](mtmv) Judge a base table column change by
the MV query it reaches (#68646)
dd456ab7a89 is described below
commit dd456ab7a894f54c6553415223c6d5c9c7d4e035
Author: yujun <[email protected]>
AuthorDate: Sat Oct 10 18:41:32 2026 +0800
[improvement](mtmv) Judge a base table column change by the MV query it
reaches (#68646)
A column change on a base table (`DROP COLUMN` / `RENAME COLUMN`) puts
every materialized view that reads the table into `SCHEMA_CHANGE`,
whether or not the column is one the view's query names. That state is
what a strict `REFRESH ... INCREMENTAL` on an IVM MV reads -- the
refresh is escalated to a whole-MV `COMPLETE` for a column the MV never
read -- and it is what stops the transparent rewrite serving a plain MV
until a refresh has run. Neither is work the change owes.
The judgement such a change needs is the view's own query, per view: a
column that is dropped or renamed either is named by that query -- and
then the query no longer analyses, which is the state the view has to be
left in -- or it is not named, and the rows the view holds are the ones
they should be. `MTMVRelationManager` already re-analyses the query for
such a change, but only to pick a more specific reason for a record it
writes either way. This PR makes that answer the criterion, for both
kinds of MV:
* Each operation says whether the query decides it
(`AlterOp#needQueryUsabilityCheck`). `DROP COLUMN` and `RENAME COLUMN`
do. A type change does not: the query keeps analysing while the meaning
of the rows computed under the old type changes, so re-analysis says
nothing for a column the query only filters or joins on. A rename of the
table itself is not asked about either, for the reason it never was: the
view's query spells the old name, so it is unanalysable by construction,
and the query check could only attribute the invalidation to the wrong
reason. `REPLACE TABLE`, a dropped table and a base view change were
never asked about, and they still invalidate every view that reads them.
* A change that has not reached the table is not asked about
(`AlterOp#hasReachedTheTable`). A schema change that is not a light one
is applied by a job, and the MV hook runs where that job may not have
run yet: the table still holds the column the change takes away, every
query still analyses against it, and an invalidation decided on that
answer would be about the table from before the change -- while the ABA
that a dropped and re-added column leaves behind is exactly what the
invalidation exists for. What is asked is whether the change has reached
the table, which is the same fact the re-analysis reads, so the two
answers cannot disagree; a change that has not been seen that way keeps
invalidating, as it did before the queries were asked at all.
Alongside it, two things the same code owed:
* `MTMV.processBaseViewChange` applied an invalidation in a second copy
of what `MTMV.alterStatus` does (state, version bump, snapshot drop). It
goes through `alterStatus` now, so an invalidation is one place to read
whatever carried it. The view path keeps applying without a record of
its own, and that is now written down where it happens: the change to
the view is what is journaled and a replay runs this hook along with it,
so a record here would apply the same invalidation twice on a follower
-- including a version bump the leader moved once.
* `MTMVState.SCHEMA_CHANGE` now says what it means, which is more than
its name: the state is "this MV needs a whole rebuild", and the comment
names the changes that set it -- for a plain MV as well as an IVM one,
and for an IVM MV alone -- and why the constant is not renamed (its name
is what is written to disk, and a constant Gson does not know reads back
as null, dropping the state of every MV loaded from such an image
instead of failing).
An entire-table `TRUNCATE TABLE` is deliberately left as it is. The
design asked for it to be recorded as a whole-MV change; the
per-partition requirement already expresses that scope (every MV
partition that reads the table is named), and the whole-MV record would
send every partition of the MV to a rebuild and reset the streams for a
change the per-partition requirement carries -- where a stream does need
that reconcile, the refresh falls back to a `COMPLETE` on its own. Both
truncate scopes are pinned by tests now.
Behaviour changed: Yes
* A base table `DROP COLUMN` / `RENAME COLUMN` no longer invalidates a
materialized view whose query does not name the column. The view keeps
its rows and stays a rewrite candidate, and a strict `REFRESH ...
INCREMENTAL` on an IVM MV is no longer escalated to a whole-MV
`COMPLETE` because of it. A column the query does name invalidates the
view as before, and so does every other base table change.
### What each column operation does, before and after
Each row names the view it is about and the column the change is about:
| Operation on the base table | The view, and the column the change is
about | Before | After |
| --- | --- | --- | --- |
| `DROP COLUMN` | `mv: SELECT k1, SUM(amount) AS total FROM t GROUP BY
k1`; the column dropped is `spare`, which this view's query does not
read | every view that reads `t` goes to `SCHEMA_CHANGE` | the change
leaves this view alone: it keeps its rows and its rewrite snapshot, and
a strict `REFRESH ... INCREMENTAL` on an IVM view is not escalated to a
whole-view `COMPLETE`. (A column the query does read -- `amount` here --
invalidates it as before) |
| `RENAME COLUMN` | the same view; the column renamed is `spare` to
`spare2` | same as the drop: every reader is invalidated | the same
judgement as the drop. Both names are held against the query, because
neither says on its own whether the query's name moved |
| `ADD COLUMN` | `mv: SELECT o.id FROM outer_t o WHERE EXISTS (SELECT 1
FROM t i WHERE i.id = o.id AND flag = 1)` where the unqualified `flag`
is bound to `outer_t`; the column added is `t.flag` | **nothing at
all**: neither `AddColumnOp` nor `AddColumnsOp` reached the MTMV hook
(`needChangeMTMVState()` was false), so the view stayed `NORMAL` holding
rows computed with `outer_t.flag` while the query had moved to `t.flag`
| invalidated: the query reaches the name the new column answers for. (A
column this query cannot reach -- `t.unrelated` -- leaves the view
alone) |
| `MODIFY COLUMN` | `mv: SELECT k1, SUM(amount) AS total FROM t GROUP BY
k1`; the column whose type changes is `amount`, which this view only
aggregates | every reader is invalidated | unchanged: every reader is
invalidated and no query is asked. The query keeps analysing while the
rows already computed under the old type change meaning |
| an ALTER mixing clauses | the same view; `spare` dropped and the type
of `amount` changed in one statement | every reader is invalidated |
unchanged: every reader is invalidated, because the query decides a
change only where every clause of the ALTER names the column it is about
|
| `REPLACE TABLE`, `DROP TABLE`, a base view change | the same view; no
column is involved -- the whole of `t` (or a view `t` reads through) is
replaced or dropped | every reader is invalidated | unchanged |
The judgement needs the change to have reached the table for its answer
to be about the table the change left. Where it has not -- a schema
change that is not a light one, whose job has not run when the hook does
-- every reader is invalidated as before and no query is asked.
### Deferred to follow-ups
Three review findings are left open on purpose, each with the
measurement that says why, and each needs something this PR does not
carry:
- **A light schema change becomes visible to a planner before the views
that read it are invalidated**
([thread](https://github.com/apache/doris/pull/68646#discussion_r4143552253),
issue #68767). The interval is between the base table's write lock being
released (where `SchemaChangeHandler.process` has already published the
column) and the MTMV hook running from `Alter`. That is a
locking/atomicity question rather than part of the query judgement:
closing it means deciding under that lock with the journal waits
deferred, or a barrier the rewrite respects -- and observers carry the
same interval through the journal order, which the smallest of those
options would not close.
- **A view definition stored before this PR keeps its `QUALIFY`
references unqualified**
([thread](https://github.com/apache/doris/pull/68646#discussion_r4214302765)).
The visitor added here fixes the definitions a view is created with; one
already in a catalog was written by the old visitor and stays exposed to
the same move, which needs re-qualifying stored definitions (a
replay/persistence design) or holding the change against `QUALIFY`
references inside the check.
- **A name that moved into an alias over another table's column is not
judged**
([thread](https://github.com/apache/doris/pull/68646#discussion_r4219437313),
issue #68841). This and the IN-subquery alias thread beside it (fixed
here, `alias_in_subquery_*`) are the same shape to the check:
`Alias.toSlot()` gives both a slot named `flag` that comes from the
column `x`, and they differ only in where the alias was defined.
Measured: widening the branch so this one is caught flips the other one
back to `SCHEMA_CHANGE`. Both are closed by comparing the binding the
view was built with against the one the change left, which changes what
the check decides on rather than adding another name it reads.
The over-invalidation findings still open in their threads (an unused
CTE producer read as a dependency, a predicate an enclosing `EXISTS`
ignores, `ORDER BY` alias precedence, an explicitly qualified reference
to another table's column) are boundaries of a name-based reading: they
cost a rebuild and a dropped snapshot rather than a wrong answer, and
they come from the same place as the third item above. Both halves of
that reading -- what it misses and what it over-reaches -- are collected
in issue #68841 for the change that judges on the binding instead.
---
.../main/java/org/apache/doris/alter/Alter.java | 42 +-
.../main/java/org/apache/doris/catalog/MTMV.java | 29 +-
.../apache/doris/mtmv/MTMVAnalyzeQueryInfo.java | 12 +-
.../org/apache/doris/mtmv/MTMVHookService.java | 35 +-
.../java/org/apache/doris/mtmv/MTMVJobManager.java | 3 +-
.../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 11 +-
.../org/apache/doris/mtmv/MTMVRefreshEnum.java | 9 +
.../org/apache/doris/mtmv/MTMVRelationManager.java | 318 +++++++++-
.../java/org/apache/doris/mtmv/MTMVService.java | 10 +-
.../trees/plans/commands/info/AddColumnOp.java | 25 +-
.../trees/plans/commands/info/AddColumnsOp.java | 28 +-
.../plans/commands/info/AlterMTMVRenameInfo.java | 2 +-
.../plans/commands/info/AlterMTMVReplaceInfo.java | 2 +-
.../nereids/trees/plans/commands/info/AlterOp.java | 42 ++
.../trees/plans/commands/info/BaseViewInfo.java | 9 +
.../trees/plans/commands/info/DropColumnOp.java | 14 +
.../trees/plans/commands/info/RenameColumnOp.java | 18 +
.../apache/doris/mtmv/MTMVRelationManagerTest.java | 31 +-
.../doris/mtmv/ivm/IvmBaselineRebuildTest.java | 295 ++++++++-
.../test_base_drop_col_multi_level_mtmv.out | 2 +-
regression-test/data/mtmv_p0/test_base_mtmv.out | 4 +-
.../test_base_rename_col_multi_level_mtmv.out | 2 +-
.../mtmv_p0/test_drop_unreferenced_column_mtmv.out | 125 ++++
..._drop_referenced_column_baseline_rebuild.groovy | 22 +-
.../test_base_drop_col_multi_level_mtmv.groovy | 4 +-
.../test_base_rename_col_multi_level_mtmv.groovy | 4 +-
.../test_drop_unreferenced_column_mtmv.groovy | 659 +++++++++++++++++++++
27 files changed, 1668 insertions(+), 89 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
index 7f482cb36ac..02afd90fc85 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
@@ -51,6 +51,7 @@ import
org.apache.doris.common.util.PropertyAnalyzer.RewriteProperty;
import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.info.TableNameInfoUtils;
import org.apache.doris.mtmv.BaseTableInfo;
+import org.apache.doris.mtmv.MTMVHookService;
import org.apache.doris.mtmv.MTMVPartitionUtil;
import org.apache.doris.mtmv.MTMVPropertyUtil;
import org.apache.doris.mtmv.MTMVRelation;
@@ -373,8 +374,29 @@ public class Alter {
throw new DdlException("Invalid alter operations: " +
currentAlterOps);
}
if (needChangeMTMVState(alterOps)) {
- Env.getCurrentEnv().getMtmvService()
- .alterTable(oldBaseTableInfo, newBaseTableInfo,
currentAlterOps.hasReplaceTableOp());
+ // Which columns an operation's effect on a view turns on is the
operation's to say, see
+ // AlterOp#queryJudgedColumnNames, and every clause of the alter
has to name them: a batch that
+ // mixes a dropped column with a type change is decided by neither
-- no query says anything
+ // about a type change -- and stays invalidated the way it was
before the queries were asked at
+ // all. Each of them also has to have reached the table. A schema
change that is not a light one
+ // is applied by a job, which may not have run where this hook
runs: the table still holds the
+ // column the change is about, every query still analyses against
it, and an invalidation
+ // decided on that answer would be about the table from before the
change. What is asked is
+ // whether the change has reached the table, which is the same
fact the re-analysis reads, so
+ // the two answers cannot disagree.
+ boolean judgedByQuery = haveReachedTheTable(alterOps, olapTable);
+ // The names of those columns go to the hook rather than a
verdict: what the judgement is about
+ // is the column, and the hook holds the query's answer against it
-- both while asking, in case
+ // the query can reach the name some other way now, and once it
has answered, in case the table
+ // is no longer the one that answered. See MTMVRelationManager.
+ MTMVHookService.QueryJudgedChange queryJudgedChange = judgedByQuery
+ ? new MTMVHookService.QueryJudgedChange(
+
alterOps.stream().map(AlterOp::queryJudgedColumnNames).flatMap(Set::stream)
+ .collect(Collectors.toSet()),
+ () -> haveReachedTheTable(alterOps, olapTable))
+ : null;
+ Env.getCurrentEnv().getMtmvService().alterTable(oldBaseTableInfo,
newBaseTableInfo,
+ currentAlterOps.hasReplaceTableOp(), queryJudgedChange);
}
olapTable.writeLock();
try {
@@ -404,6 +426,22 @@ public class Alter {
Env.getCurrentEnv().getEditLog().logModifyTableProperties(info);
}
+ /**
+ * Whether every clause of the alter names the columns it is judged on,
and every one of them has reached
+ * the table. The columns are read under the table's read lock: a light
change of another statement
+ * rebuilds the schema map, and a reader without the lock can take the map
mid rebuild, when the column
+ * this clause added is not in it yet.
+ */
+ private boolean haveReachedTheTable(List<AlterOp> alterOps, OlapTable
olapTable) {
+ olapTable.readLock();
+ try {
+ return alterOps.stream().allMatch(op ->
!op.queryJudgedColumnNames().isEmpty()
+ && op.hasReachedTheTable(olapTable));
+ } finally {
+ olapTable.readUnlock();
+ }
+ }
+
private void processAlterTableForExternalTable(
ExternalTable table, List<AlterOp> alterOps) throws UserException {
for (AlterOp alterOp : alterOps) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index bf58705f356..4a855624e46 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -237,7 +237,13 @@ public class MTMV extends OlapTable {
* lock and moves all three, but nothing here is journaled. It is what
{@code Alter#processAlterMTMV}
* calls for an {@code ALTER_STATUS} op, both live and on replay; a live
caller reaches it through
* {@link #invalidateWholeMv}, which goes the journaled way round. A new
invalidation belongs there:
- * calling this directly would leave the MV in a state a restart forgets.
+ * calling this directly would leave the MV in a state a restart forgets.
The one caller that goes
+ * direct is {@link #processBaseViewChange}, and it is not new: the alter
or the drop of the view is
+ * journaled, and a replay runs that hook along with it.
+ *
+ * <p>An image that is loaded and turns out to be unusable sets the state
on its own, see
+ * {@link #compatible} and {@link #gsonPostProcess}: there is no change
behind it to record, and nothing
+ * a replay could apply it against.
*/
public MTMVStatus alterStatus(MTMVStatus newStatus) {
writeMvLock();
@@ -251,16 +257,19 @@ public class MTMV extends OlapTable {
}
}
+ /**
+ * Invalidates the whole MV for a change to one of the base views it reads.
+ *
+ * <p>Applied through {@link #alterStatus}, so that the three things an
invalidation moves are moved in
+ * one place whatever carried it: the state the refresh reads, the version
that discards a task result
+ * computed against it, and the snapshot drop that stops the transparent
rewrite serving rows from it.
+ *
+ * <p>It records nothing, and that is not an omission: what is journaled
is the change to the view, and
+ * a replay re-runs this hook along with it. A record here would apply the
invalidation a second time on
+ * a follower -- including the version bump, which the leader moved once.
+ */
public void processBaseViewChange(String schemaChangeDetail) {
- writeMvLock();
- try {
- this.schemaChangeVersion++;
- this.status.setState(MTMVState.SCHEMA_CHANGE);
- this.status.setSchemaChangeDetail(schemaChangeDetail);
- this.refreshSnapshot = new MTMVRefreshSnapshot();
- } finally {
- writeMvUnlock();
- }
+ alterStatus(new MTMVStatus(MTMVState.SCHEMA_CHANGE,
schemaChangeDetail));
}
public boolean isIvm() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVAnalyzeQueryInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVAnalyzeQueryInfo.java
index ab31a7d3567..a137f43b225 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVAnalyzeQueryInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVAnalyzeQueryInfo.java
@@ -32,14 +32,20 @@ public class MTMVAnalyzeQueryInfo {
private Map<String, String> properties;
// set when IVM rewrite is enabled; carries normalizedPlan + aggMeta
private IvmRewriteResult ivmRewriteResult;
+ // The plan the query was analysed into. Kept for the callers that have to
ask what the query reads
+ // rather than only that it reads something; set by
MTMVPlanUtil#analyzeQueryWithSql.
+ private Plan analyzedPlan;
public MTMVAnalyzeQueryInfo(List<ColumnDefinition> columnDefinitions,
List<String> keys,
- MTMVPartitionInfo mvPartitionInfo, MTMVRelation relation,
Map<String, String> properties) {
+ MTMVPartitionInfo mvPartitionInfo, MTMVRelation relation,
Map<String, String> properties,
+ IvmRewriteResult ivmRewriteResult, Plan analyzedPlan) {
this.columnDefinitions = columnDefinitions;
this.keys = keys;
this.mvPartitionInfo = mvPartitionInfo;
this.relation = relation;
this.properties = properties;
+ this.ivmRewriteResult = ivmRewriteResult;
+ this.analyzedPlan = analyzedPlan;
}
public List<ColumnDefinition> getColumnDefinitions() {
@@ -66,8 +72,8 @@ public class MTMVAnalyzeQueryInfo {
return properties;
}
- public void setIvmRewriteResult(IvmRewriteResult ivmRewriteResult) {
- this.ivmRewriteResult = ivmRewriteResult;
+ public Plan getAnalyzedPlan() {
+ return analyzedPlan;
}
/** Convenience accessor — returns the normalized plan, or null if IVM is
not active. */
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVHookService.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVHookService.java
index 1885d10ae45..ccd1b1e6a28 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVHookService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVHookService.java
@@ -29,6 +29,8 @@ import
org.apache.doris.nereids.trees.plans.commands.info.RefreshMTMVInfo;
import org.apache.doris.nereids.trees.plans.commands.info.ResumeMTMVInfo;
import java.util.Optional;
+import java.util.Set;
+import java.util.function.BooleanSupplier;
/**
* Contains all operations that affect the mtmv
@@ -88,8 +90,39 @@ public interface MTMVHookService {
* @param oldTableInfo info before alter
* @param newTableInfo info after alter
* @param isReplace
+ * @param queryJudgedChange the change, left to the judgement of the query
of each MV that reads the
+ * table, or null when the alter is not one a
query decides
*/
- void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace);
+ void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace,
+ QueryJudgedChange queryJudgedChange);
+
+ /**
+ * A change to a base table that is left to the query of each view that
reads it.
+ *
+ * <p>It carries the two things such an answer is held against. The
columns are what the query is asked
+ * about: they are the names the change gives the table or takes away from
it, and a name is what the
+ * change moves -- a column added where the query reaches a name answers
for it from then on, and one
+ * taken away leaves the name to whatever else answers to it. Whether the
change has reached the table is
+ * asked again after the query has been analysed, because a table moves on
between the two, and the
+ * answer has to be about the table the change left rather than about a
later one.
+ */
+ class QueryJudgedChange {
+ private final Set<String> columns;
+ private final BooleanSupplier hasReachedTheTable;
+
+ public QueryJudgedChange(Set<String> columns, BooleanSupplier
hasReachedTheTable) {
+ this.columns = columns;
+ this.hasReachedTheTable = hasReachedTheTable;
+ }
+
+ public Set<String> columns() {
+ return columns;
+ }
+
+ public boolean hasReachedTheTable() {
+ return hasReachedTheTable.getAsBoolean();
+ }
+ }
/**
* Triggered when pause mtmv
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
index a1cd8db3817..9a23ca479a9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
@@ -190,7 +190,8 @@ public class MTMVJobManager implements MTMVHookService {
}
@Override
- public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace) {
+ public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace,
+ QueryJudgedChange queryJudgedChange) {
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
index aec2406a0b8..88d4e32380f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
@@ -647,12 +647,8 @@ public class MTMVPlanUtil {
Set<String> keysSet =
Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
keysSet.addAll(keys);
validateColumns(columns, keysSet, finalEnableMergeOnWrite);
- MTMVAnalyzeQueryInfo queryInfo = new MTMVAnalyzeQueryInfo(columns,
keys, mvPartitionInfo, relation,
- properties);
- if (isIvm) {
- ivmRewriteResult.ifPresent(queryInfo::setIvmRewriteResult);
- }
- return queryInfo;
+ return new MTMVAnalyzeQueryInfo(columns, keys, mvPartitionInfo,
relation, properties,
+ isIvm ? ivmRewriteResult.orElse(null) : null,
planner.getAnalyzedPlan());
}
}
@@ -1023,7 +1019,7 @@ public class MTMVPlanUtil {
}
}
- public static void ensureMTMVQueryUsable(MTMV mtmv, ConnectContext ctx)
throws JobException {
+ public static MTMVAnalyzeQueryInfo ensureMTMVQueryUsable(MTMV mtmv,
ConnectContext ctx) throws JobException {
MTMVAnalyzeQueryInfo mtmvAnalyzedQueryInfo;
try {
mtmvAnalyzedQueryInfo = MTMVPlanUtil.analyzeQueryWithSql(mtmv, ctx,
@@ -1033,6 +1029,7 @@ public class MTMVPlanUtil {
}
checkColumnIfChange(mtmv,
mtmvAnalyzedQueryInfo.getColumnDefinitions());
checkMTMVPartitionInfo(mtmv,
mtmvAnalyzedQueryInfo.getMvPartitionInfo());
+ return mtmvAnalyzedQueryInfo;
}
private static void checkMTMVPartitionInfo(MTMV mtmv, MTMVPartitionInfo
analyzedMvPartitionInfo)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshEnum.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshEnum.java
index 32c369e436f..08880ee29b4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshEnum.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshEnum.java
@@ -55,6 +55,15 @@ public class MTMVRefreshEnum {
public enum MTMVState {
INIT,
NORMAL,
+ // The state means "this MV needs a whole rebuild", which its name no
longer says on its own: it is
+ // not only a base table schema change that sets it. A dropped base
table, a replace and a change to
+ // a base view set it for a plain MV as well as an IVM one; an
entire-table truncate, and a property
+ // change that brings base table partitions back into the set an MV
maintains, do so for an IVM MV.
+ //
+ // The constant name is what is written to disk, so it is not renamed:
MTMV.status is serialized
+ // with Gson without an enum adapter, which writes the constant name
and reads an unknown one back
+ // as null -- silently dropping the state of every MV loaded from such
an image. The status an
+ // ALTER_STATUS record carries is serialized the same way.
SCHEMA_CHANGE
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
index 51360b36790..fd19d0f2c7b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
@@ -17,6 +17,7 @@
package org.apache.doris.mtmv;
+import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.MTMV;
import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.Table;
@@ -27,17 +28,28 @@ import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.job.common.TaskStatus;
import org.apache.doris.job.exception.JobException;
import org.apache.doris.job.extensions.mtmv.MTMVTask;
+import org.apache.doris.nereids.lineage.LineageInfo;
+import org.apache.doris.nereids.lineage.LineageInfoExtractor;
import org.apache.doris.nereids.rules.exploration.mv.PartitionCompensator;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.commands.info.CancelMTMVTaskInfo;
import org.apache.doris.nereids.trees.plans.commands.info.PauseMTMVInfo;
import org.apache.doris.nereids.trees.plans.commands.info.RefreshMTMVInfo;
import org.apache.doris.nereids.trees.plans.commands.info.ResumeMTMVInfo;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
import org.apache.doris.qe.ConnectContext;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
+import com.google.common.collect.SetMultimap;
import com.google.common.collect.Sets;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.logging.log4j.LogManager;
@@ -349,34 +361,237 @@ public class MTMVRelationManager implements
MTMVHookService {
// because a dropped table is the one change whose query is gone
beyond doubt. What the two record
// is the same state either way. Unlike a rename it stays an
invalidation: the table is gone for
// good, so the state is not something a later alter can make obsolete.
- processBaseTableChange(new BaseTableInfo(table), "The base table has
been deleted:", false);
+ processBaseTableChange(new BaseTableInfo(table), "The base table has
been deleted:", null);
}
/**
* update mtmv status to `SCHEMA_CHANGE`.
*
* @param isReplace
+ * @param queryJudgedColumns the names the alter gives the table or takes
away from it, which leave the
+ * judgement about each MV's state to that MV's
own query, or null when the
+ * alter is not one a query decides. The names
are carried rather than judged
+ * before the call because the judgement is
about them; see
+ * {@code AlterOp#queryJudgedColumnNames} for
which operations name one, and
+ * {@link #invalidateMvUnlessQueryHolds} for
what is asked about it. A rename
+ * of the base table names no column: it is left
to the record below, which
+ * says what the MV that keeps spelling the old
name needs to hear
*/
@Override
- public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace) {
+ public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace,
+ QueryJudgedChange queryJudgedChange) {
// when replace, need deal two table
if (isReplace) {
// REPLACE TABLE already invalidates the IVM baseline explicitly,
see Alter#processReplaceTable
- processBaseTableChange(newTableInfo.get(), "The base table has
been updated:", false);
+ processBaseTableChange(newTableInfo.get(), "The base table has
been updated:", null);
}
- boolean renamed = !isReplace && newTableInfo.isPresent()
- && !Objects.equals(oldTableInfo.getTableName(),
newTableInfo.get().getTableName());
- // A rename is the one change whose query check is skipped: the MV
query keeps spelling the old
- // name, so it is unanalyzable by construction, and the reason it
would be invalidated with --
- // "the query is no longer analyzable" -- says less than the message
this call records anyway.
- boolean checkQueryUsable = !renamed;
- processBaseTableChange(oldTableInfo, "The base table has been
updated:", checkQueryUsable);
+ processBaseTableChange(oldTableInfo, "The base table has been
updated:", queryJudgedChange);
}
/**
- * An MV's query is only as good as the base table schema it was analyzed
against. Re-analyzing the
- * MV query here (right after the alter was applied) is what detects a
changed column identity:
+ * Whether the query, as it is analysed now, reads a column of any of
these names, and reads it where
+ * the change can reach it.
+ *
+ * <p>There are two places a name is the change's to answer for. One is a
column of the table the change
+ * is about: that is the column this view's rows were computed from, and
the names are matched
+ * case-insensitively because a name is what moves. The other is a column
the query reaches across a
+ * scope boundary -- the plan records those on the Apply that stands for
the subquery, whose correlation
+ * slots are the outer columns its right side reads -- because such a name
is the scopes' to answer for
+ * rather than the query's: the nearest column to the reference answers
for it, so a column the change
+ * takes away from a scope inside leaves the name to one outside, and a
column it gives to a scope inside
+ * takes the name over. A name reached with the qualifier of another table
inside the query's own scope
+ * is neither: no later change can move it, so one to a column it does not
name is one this view's rows
+ * do not depend on.
+ */
+ private static boolean reachesAnyColumnOf(Plan plan, BaseTableInfo
baseTableInfo, Set<String> columnNames) {
+ if (plan == null) {
+ // A query whose plan was not kept is one this cannot be answered
about, and "it does" is the
+ // answer that keeps the view safe.
+ return true;
+ }
+ Set<String> names = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
+ names.addAll(columnNames);
+ LineageInfo lineage = LineageInfoExtractor.extractLineageInfo(plan);
+ for (SetMultimap<?, Expression> byType :
lineage.getDirectLineageMap().values()) {
+ if (reachesAnyColumn(byType.values(), names, baseTableInfo)) {
+ return true;
+ }
+ }
+ // The dataset predicates once, not once per output column: the
per-output copy of them the lineage
+ // also offers holds the same expressions for every column the query
produces, and scanning it would
+ // visit each of them once per column.
+ if (reachesAnyColumn(lineage.getDatasetIndirectLineageMap().values(),
names, baseTableInfo)) {
+ return true;
+ }
+ if (reachesAnyColumnOfASubquery(plan, names, baseTableInfo)) {
+ return true;
+ }
+ return reachesAnyColumnAcrossScopes(plan, lineage, names,
baseTableInfo);
+ }
+
+ /** Whether this slot is a column of this table, through whatever views
stand between the two. */
+ private static boolean isColumnOf(Slot slot, BaseTableInfo baseTableInfo) {
+ if (!(slot instanceof SlotReference)) {
+ return false;
+ }
+ return ((SlotReference) slot).getOriginalTable()
+ .map(table -> new BaseTableInfo(table).equals(baseTableInfo))
+ .orElse(false);
+ }
+
+ /**
+ * Whether a name the change is about is answered for inside a subquery,
out of that subquery's own
+ * scope.
+ *
+ * <p>This is the one place a name can move without any column the view
produces depending on it: the
+ * scope of a subquery is internal, so which column answers for a name
there changes what the query
+ * returns -- a row, or none -- while every column of the view stays the
one it was. The lineage of the
+ * view's columns does not reach it, so the scope the subquery became is
read here, expression by
+ * expression, the way the lineage is read for the view's own.
+ *
+ * <p>Two things are read. One is a value the subquery itself names -- an
expression of its own under one
+ * of these names, rather than a column of a table -- because that is what
a name the change takes away
+ * falls back to, and it decides the rows whether the subquery is a
predicate or a value. It is read only
+ * where the table the change is about is one that subquery reads: what a
name falls back to is what the
+ * scope that answered for it holds, so a scope that does not read the
table holds nothing for the name
+ * and one of its own is one the change never reached. The other is a
column of the table the change is
+ * about, which decides the rows only when the subquery's output is one
the query reads: an EXISTS tests
+ * the rows of its subquery and not what it projects, so a name it
projects and never compares is one
+ * this view's rows do not depend on.
+ *
+ * <p>Each scope is read on its own. A subquery inside one of these is a
scope of its own, and it is
+ * judged where it is read and not as a part of its enclosing one: its
projection is held only where
+ * that scope's own output is read, so an EXISTS inside an IN is still not
compared by the IN.
+ */
+ private static boolean reachesAnyColumnOfASubquery(Plan plan, Set<String>
names,
+ BaseTableInfo baseTableInfo) {
+ for (LogicalApply<?, ?> apply :
plan.<LogicalApply>collectToList(LogicalApply.class::isInstance)) {
+ List<Plan> itsOwnScope = ownScopeOf(apply);
+ boolean outputDecidesRows = !apply.isExist();
+ boolean isOneOfItsTables = readsAnyTableOf(itsOwnScope,
baseTableInfo);
+ for (Plan node : itsOwnScope) {
+ for (Expression expression : node.getExpressions()) {
+ if ((isOneOfItsTables &&
readsAnyNameTheSubqueryAnswersFor(expression, names))
+ || (outputDecidesRows &&
reachesAnyColumn(expression, names, baseTableInfo))) {
+ return true;
+ }
+ }
+ }
+ }
+ return false;
+ }
+
+ /**
+ * The nodes of a subquery that belong to the scope this Apply stands for:
its right side, with the
+ * right side of every subquery nested in it left out.
+ *
+ * <p>A nested Apply is a scope of its own and is read as one, so its
right side belongs to that read
+ * rather than to this one; the relation it is asked about is the
enclosing scope's own and stays.
+ */
+ private static List<Plan> ownScopeOf(LogicalApply<?, ?> apply) {
+ List<Plan> itsOwnScope = Lists.newArrayList();
+ collectItsOwnScope((Plan) apply.right(), itsOwnScope);
+ return itsOwnScope;
+ }
+
+ private static void collectItsOwnScope(Plan node, List<Plan> itsOwnScope) {
+ itsOwnScope.add(node);
+ if (node instanceof LogicalApply) {
+ collectItsOwnScope(((LogicalApply<?, ?>) node).left(),
itsOwnScope);
+ return;
+ }
+ for (Plan child : node.children()) {
+ collectItsOwnScope(child, itsOwnScope);
+ }
+ }
+
+ /** Whether one of these nodes reads this table, through whatever views
stand between the two. */
+ private static boolean readsAnyTableOf(List<Plan> itsOwnScope,
BaseTableInfo baseTableInfo) {
+ return itsOwnScope.stream().anyMatch(node -> node instanceof
LogicalCatalogRelation
+ && new BaseTableInfo(((LogicalCatalogRelation)
node).getTable()).equals(baseTableInfo));
+ }
+
+ /**
+ * Whether this expression reads a value the subquery answers for itself,
under one of these names: a
+ * slot of the subquery's own -- an alias or a value it computed -- rather
than a column of a table.
+ *
+ * <p>Read rather than merely named, because an expression of the subquery
carrying one of these names
+ * says nothing on its own: a subquery that names a `flag` of its own
while no reference in it resolves
+ * to that name is one whose rows the change cannot reach, and one that
reads the name it names is where
+ * a reference that answered for the changed column falls back to.
+ */
+ private static boolean readsAnyNameTheSubqueryAnswersFor(Expression
expression, Set<String> names) {
+ for (Slot slot : expression.getInputSlots()) {
+ if (names.contains(slot.getName()) && !isColumnOfATable(slot)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** Whether this slot is a column of some table, or a value produced
inside the query. */
+ private static boolean isColumnOfATable(Slot slot) {
+ return slot instanceof SlotReference && ((SlotReference)
slot).getOriginalTable().isPresent();
+ }
+
+ /** Whether this expression reads a column of one of these names from this
table. */
+ private static boolean reachesAnyColumn(Expression expression, Set<String>
names,
+ BaseTableInfo baseTableInfo) {
+ return reachesAnyColumn(ImmutableList.of(expression), names,
baseTableInfo);
+ }
+
+ /**
+ * Whether any of these expressions reads a column of one of these names
from this table.
+ *
+ * <p>The name is the one the table gives the column, not the one the slot
is called: an alias renames
+ * what a slot is known by without moving the column it comes from, so a
slot named after a column it
+ * does not read is not reading that column, and a slot renamed to
something else is still reading its
+ * own. A change is a change to the column under the name the table gives
it, so this is the name the
+ * change is held against.
+ */
+ private static boolean reachesAnyColumn(Collection<? extends Expression>
expressions, Set<String> names,
+ BaseTableInfo baseTableInfo) {
+ for (Expression expression : expressions) {
+ for (Slot slot : expression.getInputSlots()) {
+ if (isColumnOf(slot, baseTableInfo) &&
names.contains(nameOfTheColumnItReads(slot))) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ /** The name the table gives the column this slot reads, whatever the slot
itself is called. */
+ private static String nameOfTheColumnItReads(Slot slot) {
+ if (!(slot instanceof SlotReference)) {
+ return slot.getName();
+ }
+ return ((SlotReference)
slot).getOriginalColumn().map(Column::getName).orElseGet(slot::getName);
+ }
+
+ /**
+ * Whether the query resolves a column of one of these names across a
scope boundary, which is a name
+ * the change can move whatever the query writes it against.
+ */
+ private static boolean reachesAnyColumnAcrossScopes(Plan plan, LineageInfo
lineage, Set<String> names,
+ BaseTableInfo baseTableInfo) {
+ // A name is only one the change can move if the table it is about is
one the query reads at all.
+ boolean isOneOfItsTables = lineage.getTableLineageSet().stream()
+ .anyMatch(table -> new
BaseTableInfo(table).equals(baseTableInfo));
+ if (!isOneOfItsTables) {
+ return false;
+ }
+ return plan.anyMatch(node -> node instanceof LogicalApply
+ && ((LogicalApply<?, ?>) node).getCorrelationSlot().stream()
+ .anyMatch(slot -> names.contains(slot.getName())));
+ }
+
+ /**
+ * An MV's query is only as good as the base table schema it was analyzed
against, and a query that
+ * analyses is not enough on its own: a name it reaches a column by can
move to another column, and the
+ * table can move on between the analysis and the answer. Re-analyzing the
MV query here (right after
+ * the alter was applied) is what detects a changed column identity:
* dropping or renaming a column the MV uses makes the query unanalyzable,
and a column re-added with
* the same name is a different column, so pre-existing rows read its
default value instead.
*
@@ -385,18 +600,23 @@ public class MTMVRelationManager implements
MTMVHookService {
* rows computed under the old column epoch. Invalidating the MV is what
keeps that from being
* reported as current.
*
+ * <p>The check is the criterion, not just the reason for the record: a
column the query does not name
+ * is one this change leaves the MV's rows alone for, so nothing is
invalidated for it. It is a whole
+ * query that is analysed, not a column that is looked up: what the MV can
no longer be computed from
+ * is what the analysis refuses, wherever in the query it stood.
+ *
* <p>Every MV is checked, not only an IVM one: whether the query still
analyzes is a property of
* the MV and of the base table it reads, not of how the MV refreshes, and
the invalidation is the
* same one a change to that table records. What an IVM MV has on top of
it is a per-partition
* requirement, and that is decided elsewhere, from a query that analyzed.
*
- * @return whether the MV was invalidated. That is the whole record for
this change: the invalidation
- * carries the reason, and the caller has nothing left to write --
a second record would land
- * on the same state, and MTMVStatus#updateStateAndDetail would
overwrite the detail with the
- * blunter "the base table has been updated", which is what
knowing the query is unusable is
- * for. It would also bump the version and drop the snapshot twice
for one change.
+ * @return whether the MV was invalidated. False is the answer for a
change that reaches neither the
+ * query nor the rows it computed, and it is the whole record for
that change: there is nothing to
+ * write, and writing the generic "the base table has been
updated" anyway would stand for a
+ * rebuild the MV does not owe.
*/
- private boolean invalidateMvIfQueryUnusable(BaseTableInfo baseTableInfo,
Table mvTable) {
+ private boolean invalidateMvUnlessQueryHolds(BaseTableInfo baseTableInfo,
Table mvTable,
+ QueryJudgedChange queryJudgedChange) {
if (!(mvTable instanceof MTMV)) {
return false;
}
@@ -404,9 +624,10 @@ public class MTMVRelationManager implements
MTMVHookService {
// Analyse in a context owned by this check, never the session that
issued the alter: the check
// must not disturb the running statement, and it has to work on
threads that have no session.
// Setting a thread local is how a context is made current, so restore
the previous one.
+ MTMVAnalyzeQueryInfo analyzedQueryInfo;
ConnectContext previousCtx = ConnectContext.get();
try {
- MTMVPlanUtil.ensureMTMVQueryUsable(mtmv,
+ analyzedQueryInfo = MTMVPlanUtil.ensureMTMVQueryUsable(mtmv,
MTMVPlanUtil.createMTMVContext(mtmv,
MTMVPlanUtil.DISABLE_RULES_WHEN_RUN_MTMV_TASK));
} catch (Exception e) {
LOG.info("Invalidate MV, the MV query is no longer usable.
baseTable={}, mtmv={}, reason={}",
@@ -420,6 +641,38 @@ public class MTMVRelationManager implements
MTMVHookService {
ConnectContext.remove();
}
}
+ // A query that still analyses has not necessarily kept its meaning: a
name can move. The column a
+ // query reaches a name by is the nearest one to it in the query's
scopes, so a column the change
+ // takes away leaves the name to whatever else answers to it -- an
unqualified name inside a subquery
+ // falls back to a correlated outer one, or to one of a table joined
there -- and a column the change
+ // adds can answer for the name from then on. Either way the query
produces the columns it always
+ // produced while their rows come from elsewhere, and the view's rows
are no longer the ones the
+ // query computes. What the query reads is read from the analysed
query's lineage, which names the
+ // columns the query really reaches -- through its projections,
filters, joins and aggregation, and
+ // through whatever views stand between them -- so an alias or a
string that happens to read the same
+ // is not one of them, and one reached inside a view is.
+ if (reachesAnyColumnOf(analyzedQueryInfo.getAnalyzedPlan(),
baseTableInfo, queryJudgedChange.columns())) {
+ LOG.info("Invalidate MV, the MV query reads a column the change is
about. "
+ + "baseTable={}, columns={}, mtmv={}",
baseTableInfo, queryJudgedChange.columns(),
+ mtmv.getName());
+ mtmv.invalidateWholeMv("The MV query reads a column the change is
about: " + baseTableInfo)
+ .await();
+ return true;
+ }
+ // And the answer has to be about the table the change left. The
analysis reads the table as it is
+ // now, and a light change is applied before this hook runs, so a
table that has moved on since --
+ // a column taken away and added back under the same name, say --
would have been analysed in that
+ // later state: the same shapes, a different column. Asked of the
change itself rather than of the
+ // columns, because what the table should hold is what that change
asked for, and the answer is the
+ // one the analysis was given about only while it still holds.
+ if (!queryJudgedChange.hasReachedTheTable()) {
+ LOG.info("Invalidate MV, the table is no longer the one the change
was applied to. "
+ + "baseTable={}, columns={}, mtmv={}",
baseTableInfo, queryJudgedChange.columns(),
+ mtmv.getName());
+ mtmv.invalidateWholeMv("The table is no longer the one the change
was applied to: "
+ + baseTableInfo).await();
+ return true;
+ }
return false;
}
@@ -483,13 +736,14 @@ public class MTMVRelationManager implements
MTMVHookService {
}
/**
- * Puts every MV that reads this base table into {@code SCHEMA_CHANGE}.
+ * Puts every MV that reads this base table into {@code SCHEMA_CHANGE} --
or, for the changes that ask
+ * for it, every MV that change does not leave alone.
*
- * @param checkQueryUsable whether to re-analyze each MV's query first; see
- * {@link #invalidateMvIfQueryUnusable}
+ * @param queryJudgedChange the change, left to each MV's own query, or
null when the alter is not one a
+ * query decides; see {@link
#invalidateMvUnlessQueryHolds}
*/
private void processBaseTableChange(BaseTableInfo baseTableInfo, String
msgPrefix,
- boolean checkQueryUsable) {
+ QueryJudgedChange queryJudgedChange) {
Set<BaseTableInfo> mtmvsByBaseTable =
getMtmvsByBaseTableOneLevelAndFromView(baseTableInfo);
if (CollectionUtils.isEmpty(mtmvsByBaseTable)) {
return;
@@ -502,10 +756,20 @@ public class MTMVRelationManager implements
MTMVHookService {
LOG.warn(e);
continue;
}
- if (checkQueryUsable && invalidateMvIfQueryUnusable(baseTableInfo,
mvTable)) {
- // Invalidated with the reason, which is the more specific of
the two messages and the one
- // this change is worth recording: the state is the same one
the generic record below
- // would set, so writing it too would only bury the reason.
+ if (queryJudgedChange != null) {
+ // The change is left to each view's own query, which is asked
and whose answer is held to
+ // the two things that can move under it -- see
invalidateMvUnlessQueryHolds. A view the
+ // change leaves alone is not invalidated: that would discard
the result of a refresh
+ // running against it on the strength of a change that never
reached it. Nothing else is
+ // recorded here either -- the invalidation carries its
reason, and a second record would
+ // land on the same state with the blunter "the base table has
been updated", having bumped
+ // the version and dropped the snapshot a second time for one
change.
+ if (invalidateMvUnlessQueryHolds(baseTableInfo, mvTable,
queryJudgedChange)) {
+ LOG.info("Invalidated MV, baseTable={}, mv={}",
baseTableInfo, mvTable.getName());
+ } else {
+ LOG.info("The change leaves the MV alone, nothing to
invalidate. baseTable={}, mv={}",
+ baseTableInfo, mvTable.getName());
+ }
continue;
}
if (!(mvTable instanceof MTMV)) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
index c9fb6005793..85fa48f6edb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
@@ -113,13 +113,15 @@ public class MTMVService implements EventListener {
}
}
- public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace) {
+ public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo>
newTableInfo, boolean isReplace,
+ MTMVHookService.QueryJudgedChange queryJudgedChange) {
Objects.requireNonNull(oldTableInfo, "oldTableInfo can not be null");
Objects.requireNonNull(newTableInfo, "newTableInfo can not be null");
- LOG.info("alterTable, oldTableInfo: {}, newTableInfo: {}, isReplace:
{}", oldTableInfo, newTableInfo,
- isReplace);
+ LOG.info("alterTable, oldTableInfo: {}, newTableInfo: {}, isReplace:
{}, queryJudgedColumns: {}",
+ oldTableInfo, newTableInfo, isReplace,
+ queryJudgedChange == null ? null :
queryJudgedChange.columns());
for (MTMVHookService mtmvHookService : hooks.values()) {
- mtmvHookService.alterTable(oldTableInfo, newTableInfo, isReplace);
+ mtmvHookService.alterTable(oldTableInfo, newTableInfo, isReplace,
queryJudgedChange);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java
index eaef1483036..ac034bb2c06 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java
@@ -36,6 +36,7 @@ import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.Sets;
+import java.util.Collections;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
@@ -119,7 +120,29 @@ public class AddColumnOp extends AlterTableOp {
@Override
public boolean needChangeMTMVState() {
- return false;
+ // The name the column adds can be the one a view's query reaches a
column by, and the column it
+ // reaches is the nearest one in the query's scopes: one added there
answers for the name from then
+ // on, and the rows the view holds are the ones of the column that
used to. That is the view's query
+ // to judge, see {@link AlterOp#queryJudgedColumnNames}.
+ return true;
+ }
+
+ @Override
+ public boolean hasReachedTheTable(OlapTable table) {
+ // The column has to be in the table: a change that is not a light one
is applied by a job, and a
+ // query judged before that job has run is judged against the table
from before the change -- which
+ // is the one that cannot be answered about at all.
+ for (String columnName : queryJudgedColumnNames()) {
+ if (table.getColumn(columnName) == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public Set<String> queryJudgedColumnNames() {
+ return Collections.singleton(getColumnDef().getName());
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java
index 9ff326ffb38..086d61bbe97 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java
@@ -27,10 +27,12 @@ import org.apache.doris.common.UserException;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import java.util.Set;
/**
* AddColumnsOp
@@ -108,7 +110,31 @@ public class AddColumnsOp extends AlterTableOp {
@Override
public boolean needChangeMTMVState() {
- return false;
+ // As in {@link AddColumnOp}: a name a view's query reaches a column
by can be taken over by one of
+ // these, and that is the view's query to judge.
+ return true;
+ }
+
+ @Override
+ public boolean hasReachedTheTable(OlapTable table) {
+ // The column has to be in the table: a change that is not a light one
is applied by a job, and a
+ // query judged before that job has run is judged against the table
from before the change -- which
+ // is the one that cannot be answered about at all.
+ for (String columnName : queryJudgedColumnNames()) {
+ if (table.getColumn(columnName) == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public Set<String> queryJudgedColumnNames() {
+ Set<String> names = Sets.newHashSetWithExpectedSize(columns.size());
+ for (Column column : columns) {
+ names.add(column.getName());
+ }
+ return names;
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVRenameInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVRenameInfo.java
index 43cd1447e76..e91e50b8870 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVRenameInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVRenameInfo.java
@@ -73,6 +73,6 @@ public class AlterMTMVRenameInfo extends AlterMTMVInfo {
BaseTableInfo oldTableInfo = new BaseTableInfo(table);
Env.getCurrentEnv().renameTable(db, table, newName.getTbl());
BaseTableInfo newTableInfo = new BaseTableInfo(table);
- Env.getCurrentEnv().getMtmvService().alterTable(oldTableInfo,
Optional.of(newTableInfo), false);
+ Env.getCurrentEnv().getMtmvService().alterTable(oldTableInfo,
Optional.of(newTableInfo), false, null);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVReplaceInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVReplaceInfo.java
index 3467b60015c..fcd315494f5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVReplaceInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterMTMVReplaceInfo.java
@@ -94,6 +94,6 @@ public class AlterMTMVReplaceInfo extends AlterMTMVInfo {
MTMV newMtmv = (MTMV) db.getTableOrDdlException(newName,
TableType.MATERIALIZED_VIEW);
Env.getCurrentEnv().getAlterInstance().processReplaceTable(db, mtmv,
newName, swapTable, true);
Env.getCurrentEnv().getMtmvService()
- .alterTable(new BaseTableInfo(mtmv), Optional.of(new
BaseTableInfo(newMtmv)), true);
+ .alterTable(new BaseTableInfo(mtmv), Optional.of(new
BaseTableInfo(newMtmv)), true, null);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterOp.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterOp.java
index b77d8386744..4d0639d1fc8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterOp.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AlterOp.java
@@ -18,10 +18,13 @@
package org.apache.doris.nereids.trees.plans.commands.info;
import org.apache.doris.alter.AlterOpType;
+import org.apache.doris.catalog.OlapTable;
import org.apache.commons.lang3.NotImplementedException;
+import java.util.Collections;
import java.util.Map;
+import java.util.Set;
/**
* AlterOp
@@ -42,6 +45,45 @@ public abstract class AlterOp {
public abstract boolean needChangeMTMVState();
+ /**
+ * The columns this operation gives the table or takes away from it, for
the operations whose effect on
+ * a view is decided by that view's own query; empty for every operation
whose effect the query does not
+ * decide.
+ *
+ * <p>Marked on the operations whose only way of reaching a view is
through a column, in either
+ * direction. A column that is dropped or renamed is either one the view's
query reaches -- and then the
+ * view cannot be computed from the table as it now stands, or is computed
from another column that
+ * answers to the same name -- or it is one the query never reaches, and
the view's rows are the ones
+ * they already were. A column that is added can move a name the query
reaches: the name is answered by
+ * the column nearest to it in the query's scopes, so one added there
takes it away from wherever the
+ * query reached it. A type change is neither: the query keeps analysing,
and re-analysis compares the
+ * columns the query produces, so a column the query only filters or joins
on can change what the view's
+ * rows mean while re-analysis reports nothing.
+ *
+ * <p>The names are what the judgement is about, rather than the operation
as a whole: they are the
+ * columns the view's query is asked about, and what the answer is checked
against once it has been
+ * given. See {@code MTMVRelationManager}, which asks the query and holds
the answer to these names.
+ *
+ * <p>Default empty: the operation invalidates every view that reads the
table.
+ */
+ public Set<String> queryJudgedColumnNames() {
+ return Collections.emptySet();
+ }
+
+ /**
+ * Whether the change this operation asks for has reached the table yet.
+ *
+ * <p>It is what a judgement about the materialized views that read the
table has to wait for. A schema
+ * change is applied by a job unless it is a light one, and the hook that
judges the views runs where
+ * such a job may not have run yet -- the table still holds the column the
change takes away, and
+ * re-analysing a view's query against it would answer for the table from
before the change.
+ *
+ * <p>Default true: an operation whose effect is in the statement itself
has nothing to wait for.
+ */
+ public boolean hasReachedTheTable(OlapTable table) {
+ return true;
+ }
+
// Whether this alter operation is allowed on tables that enable row
binlog.
// Default is false, and only operations that are explicitly marked as safe
// for row binlog should override this to return true.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
index 3a0d8263a28..7e2459fe1ba 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
@@ -60,6 +60,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalQualify;
import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
@@ -392,6 +393,14 @@ public class BaseViewInfo {
return project.child().accept(this, context);
}
+ @Override
+ public Void visitLogicalQualify(LogicalQualify<? extends Plan>
qualify, StatementContext context) {
+ for (Expression expr : qualify.getConjuncts()) {
+ expr.accept(SlotDealer.INSTANCE, context);
+ }
+ return qualify.child().accept(this, context);
+ }
+
@Override
public Void visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat,
StatementContext context) {
for (Expression expr : repeat.getOutputExpressions()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java
index f9a3444cbe0..13278345ef0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java
@@ -35,6 +35,7 @@ import org.apache.doris.qe.ConnectContext;
import com.google.common.base.Strings;
+import java.util.Collections;
import java.util.Map;
import java.util.Set;
@@ -172,6 +173,19 @@ public class DropColumnOp extends AlterTableOp {
return true;
}
+ @Override
+ public Set<String> queryJudgedColumnNames() {
+ return Collections.singleton(getColName());
+ }
+
+ @Override
+ public boolean hasReachedTheTable(OlapTable table) {
+ // A path into a column leaves the column itself in place, so what the
table holds is the container
+ // of the part this asks about: dropping the part cannot be told apart
from a change that has not
+ // been applied, and that is the answer which keeps the views
invalidated.
+ return !columnPath.isNested() && table.getColumn(getColName()) == null;
+ }
+
@Override
public boolean allowOpRowBinlog() {
// Drop column is considered safe for row binlog tables after whitelist
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java
index 15270360535..bd3374aeaba 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.trees.plans.commands.info;
import org.apache.doris.alter.AlterOpType;
import org.apache.doris.analysis.ColumnPath;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.OlapTable;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.FeNameFormat;
import org.apache.doris.common.UserException;
@@ -27,8 +28,10 @@ import org.apache.doris.common.util.SqlUtils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.base.Strings;
+import com.google.common.collect.Sets;
import java.util.Map;
+import java.util.Set;
/**
* RenameColumnOp
@@ -97,6 +100,21 @@ public class RenameColumnOp extends AlterTableOp {
return true;
}
+ @Override
+ public Set<String> queryJudgedColumnNames() {
+ // Both names: the column the query used to reach under the old one,
and the name the new one takes
+ // over, which a query that reaches that name from anywhere else can
be bound to from now on.
+ return Sets.newHashSet(getColName(), newColName);
+ }
+
+ @Override
+ public boolean hasReachedTheTable(OlapTable table) {
+ // The name the query spells is what a rename takes away, so a table
that no longer has it is one
+ // the change has reached -- the question a drop asks as well. A path
into a column is the case this
+ // cannot answer for, as it is for a drop.
+ return !columnPath.isNested() && table.getColumn(getColName()) == null;
+ }
+
@Override
public String toSql() {
return "RENAME COLUMN " + columnPath.toSql() + " " +
SqlUtils.getIdentSql(newColName);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java
index b58137c9564..5109d399fb1 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java
@@ -188,7 +188,8 @@ public class MTMVRelationManagerTest {
* a concurrent task result, leaving the follower in SCHEMA_CHANGE where
this FE ended NORMAL.
*
* <p>A rename is the shortest way into this path: it skips the query
check, so what the hook does is
- * exactly the record under test.
+ * exactly the record under test. The rename is passed with the check
asked for, which it cannot be
+ * judged by -- the MV query keeps spelling the old name -- and the flag
does not reach it.
*/
@Test
public void testABaseTableRenameIsRecordedThroughTheInvalidation() {
@@ -201,7 +202,8 @@ public class MTMVRelationManagerTest {
try (MockedStatic<MTMVUtil> util = Mockito.mockStatic(MTMVUtil.class))
{
util.when(() ->
MTMVUtil.getTable(Mockito.any(BaseTableInfo.class))).thenReturn(mtmv);
- manager.alterTable(t3, Optional.of(t4), false);
+ // A rename of the table names no column, so the alter does not
hand the judgement to a query.
+ manager.alterTable(t3, Optional.of(t4), false, null);
}
ArgumentCaptor<String> detail = ArgumentCaptor.forClass(String.class);
@@ -210,4 +212,29 @@ public class MTMVRelationManagerTest {
detail.getValue());
Mockito.verify(editLogItem).await();
}
+
+ /**
+ * A change the alter does not hand to the queries is invalidated as a
base table change was before the
+ * queries were asked at all. The alter says so by naming no column: it
does that where every clause of
+ * it is not one a query decides, and where the schema change has not
reached the table yet -- asking
+ * then would answer for the table from before the change.
+ */
+ @Test
+ public void testAChangeTheQueryCannotBeAskedAboutInvalidates() {
+ MTMVRelationManager manager = new MTMVRelationManager();
+ manager.refreshMTMVCache(new MTMVRelation(Sets.newHashSet(t3),
Sets.newHashSet(t3),
+ Sets.newHashSet(t3), Sets.newHashSet(), Sets.newHashSet()),
mv1);
+ MTMV mtmv = Mockito.mock(MTMV.class);
+ EditLogItem editLogItem = Mockito.mock(EditLogItem.class);
+
Mockito.when(mtmv.invalidateWholeMv(Mockito.anyString())).thenReturn(editLogItem);
+ try (MockedStatic<MTMVUtil> util = Mockito.mockStatic(MTMVUtil.class))
{
+ util.when(() ->
MTMVUtil.getTable(Mockito.any(BaseTableInfo.class))).thenReturn(mtmv);
+
+ // The same table on both sides: this is not a rename, so what the
alter names alone decides.
+ manager.alterTable(t3, Optional.of(t3), false, null);
+ }
+
+ Mockito.verify(mtmv).invalidateWholeMv(Mockito.anyString());
+ Mockito.verify(editLogItem).await();
+ }
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
index 7f6b583b8d8..0cf0f85a286 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
@@ -17,6 +17,7 @@
package org.apache.doris.mtmv.ivm;
+import org.apache.doris.analysis.ColumnPath;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.MTMV;
@@ -43,6 +44,8 @@ import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVState;
import org.apache.doris.mtmv.MTMVRefreshPartitionSnapshot;
import org.apache.doris.mtmv.MTMVRelation;
import org.apache.doris.mtmv.MTMVStatus;
+import org.apache.doris.nereids.trees.plans.commands.info.DropColumnOp;
+import org.apache.doris.nereids.trees.plans.commands.info.RenameColumnOp;
import org.apache.doris.persist.AlterMTMV;
import org.apache.doris.persist.DropPartitionInfo;
import org.apache.doris.persist.EditLog;
@@ -88,6 +91,54 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
Assertions.assertEquals(MTMVState.SCHEMA_CHANGE,
getMtmv(db).getStatus().getState());
}
+ /**
+ * A truncated partition is placed on the MV partitions that read it, the
way a dropped partition is:
+ * the requirement is what carries the change, not the state, and every
other partition keeps catching
+ * up incrementally.
+ */
+ @Test
+ public void testTruncateAPartitionStaysOnThePartitionsThatReadIt() throws
Exception {
+ String db = "ivm_truncate_one_partition";
+ createPartitionedIvmTableAndPartitionedMv(db);
+ MTMV mtmv = getMtmv(db);
+ Set<String> expected = mvPartitionsWithSameRange(mtmv,
getBaseTable(db), "p202001");
+ Assertions.assertEquals(1, expected.size());
+
+ alignStatesOf(mtmv);
+ executeSql("TRUNCATE TABLE ivm_base PARTITION(p202001)");
+
+ Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
+ for (String partitionName : mtmv.getPartitionNames()) {
+ long expectedLatest = expected.contains(partitionName) ? 2 : 1;
+ Assertions.assertEquals(expectedLatest,
+
mtmv.getPartitionStates().get(partitionName).getLatestEpoch());
+ }
+ }
+
+ /**
+ * An entire-table truncate is recorded the same way, and this is the case
that says the scope of the
+ * change is not what decides: every MV partition reads a partition that
was replaced, so every one of
+ * them has its requirement raised -- and the state, which is the coarser
record, is not what carries
+ * it. Recording it as a whole-MV change instead would send every
partition of the MV to a rebuild and
+ * reset the streams, for a change whose scope the per-partition
requirement already expresses.
+ */
+ @Test
+ public void testTruncateTableStaysOnThePartitionsThatReadIt() throws
Exception {
+ String db = "ivm_truncate_entire_table";
+ createPartitionedIvmTableAndPartitionedMv(db);
+ MTMV mtmv = getMtmv(db);
+ Assertions.assertEquals(2, mtmv.getPartitionNames().size());
+
+ alignStatesOf(mtmv);
+ executeSql("TRUNCATE TABLE ivm_base");
+
+ Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
+ for (String partitionName : mtmv.getPartitionNames()) {
+ Assertions.assertEquals(2,
+
mtmv.getPartitionStates().get(partitionName).getLatestEpoch());
+ }
+ }
+
@Test
public void testTruncatePartitionMarksBaselineRebuild() throws Exception {
String db = "ivm_broken_truncate_partition";
@@ -130,17 +181,19 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
createPartitionedIvmTableAndMv(db);
MTMV mtmv = getMtmv(db);
- // ivm_mv selects dt, k1, v1. Dropping a column it does not use must
leave the baseline alone: no
- // partition's requirement is raised, so no partition is sent to a
rebuild. The MV state is not the
- // witness here -- a column change puts any MV into SCHEMA_CHANGE
through the shared base-table hook,
- // IVM or not (see testRenameTableMarksBaselineRebuild), so telling a
referenced column from an
- // unreferenced one is that hook's criterion to refine and not this
one's.
+ // ivm_mv selects dt, k1, v1. Dropping a column it does not use leaves
the MV where it is: no
+ // partition's requirement is raised, and the MV is not invalidated
either. The criterion is the
+ // MV's own query -- a column the query does not name leaves it
analyzable, so there is no rebuild
+ // for the state to stand for, and writing one would discard the
result of a refresh running
+ // against rows that are still the rows they should be.
alignStatesOf(mtmv);
+ long versionBefore = mtmv.getSchemaChangeVersion();
Map<String, Long> before = latestEpochsOf(mtmv);
executeSql("ALTER TABLE ivm_base ADD COLUMN spare int");
executeSql("ALTER TABLE ivm_base DROP COLUMN spare");
Assertions.assertEquals(before, latestEpochsOf(mtmv),
"a column the MV does not use must not raise any partition's
requirement");
+ assertNotInvalidated(mtmv, versionBefore, "a column the MV does not
use must not invalidate the MV");
// Dropping a column the MV uses makes the MV query unanalyzable: the
change is metadata-only
// and emits no binlog, so an incremental refresh would silently keep
the rows of the old
@@ -224,19 +277,204 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
MTMV mtmv = getMtmv(db);
Assertions.assertFalse(mtmv.isIvm());
- // A column this MV does not read leaves its query alone, so the MV is
put into SCHEMA_CHANGE for
- // the ordinary reason -- which is what the shared hook does for any
column change, IVM or not --
- // and not because its query went away. The detail is what tells the
two apart here, because the
- // state is the same either way.
+ // A column this MV does not read leaves its query alone, so the MV is
left alone too, IVM or not:
+ // the state is a rebuild the next refresh owes, and there is none to
owe. What an IVM MV has on top
+ // of the state is its epoch state, and that stays where it was as
well.
+ long versionBefore = mtmv.getSchemaChangeVersion();
executeSql("ALTER TABLE ivm_base ADD COLUMN spare int");
executeSql("ALTER TABLE ivm_base DROP COLUMN spare");
+ assertNotInvalidated(mtmv, versionBefore, "a column the MV does not
read must not invalidate it");
+
+ // A column it does read takes the query away, and that is what the MV
is invalidated with.
+ executeSql("ALTER TABLE ivm_base DROP COLUMN v1");
+ Assertions.assertEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
+ assertUnanalyzableDetail(mtmv);
+ }
+
+ /**
+ * A renamed column is judged the same way a dropped one is, and the
operation says so for itself: a row
+ * binlog table refuses a column rename outright, so this is a plain MV's
path, and the re-analysis is
+ * the answer for it -- the query names the old name, or it does not name
the column at all.
+ */
+ @Test
+ public void testARenamedColumnIsJudgedByTheQuery() throws Exception {
+ String db = "ivm_renamed_column";
+ createPartitionedTableWithoutRowBinlog(db);
+ createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n"
+ + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n"
+ + "DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1')\n"
+ + "AS SELECT dt, k1, v1 FROM ivm_base");
+ MTMV mtmv = getMtmv(db);
+ Assertions.assertFalse(mtmv.isIvm());
+
+ executeSql("ALTER TABLE ivm_base ADD COLUMN spare int");
+ long versionBefore = mtmv.getSchemaChangeVersion();
+ executeSql("ALTER TABLE ivm_base RENAME COLUMN spare spare_renamed");
+ assertNotInvalidated(mtmv, versionBefore, "a column the MV does not
read must not invalidate it");
+
+ executeSql("ALTER TABLE ivm_base RENAME COLUMN v1 v1_renamed");
+ Assertions.assertEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
+ assertUnanalyzableDetail(mtmv);
+ }
+
+ /**
+ * A column change is only judged by the queries once the table has it.
The query is analysed against the
+ * table as it is when the hook runs, so a schema change that has not been
applied yet leaves the column
+ * in place and the answer would be about the table from before the change
-- which is why the alter asks
+ * the operation instead of trusting its own submission. Pinned here: the
answer flips when the change
+ * lands, and a path into a column is one the question cannot be answered
for.
+ */
+ @Test
+ public void testAColumnChangeIsJudgedOnlyOnceTheTableHasIt() throws
Exception {
+ String db = "ivm_column_change_reached";
+ createPartitionedTableWithoutRowBinlog(db);
+ OlapTable baseTable = getBaseTable(db);
+
+ RenameColumnOp pendingRename = new RenameColumnOp("v1", "v1_renamed");
+ Assertions.assertFalse(pendingRename.hasReachedTheTable(baseTable),
+ "the name the query spells is still in the table, so the
rename has not reached it");
+ DropColumnOp pendingDrop = new DropColumnOp("v1", null, null);
+ Assertions.assertFalse(pendingDrop.hasReachedTheTable(baseTable),
+ "the column is still in the table, so the drop has not reached
it");
+
+ executeSql("ALTER TABLE ivm_base RENAME COLUMN v1 v1_renamed");
+
+
Assertions.assertTrue(pendingRename.hasReachedTheTable(getBaseTable(db)),
+ "the old name is gone from the table, so the rename has
reached it");
+
+ DropColumnOp dropAfterRename = new DropColumnOp("v1_renamed", null,
null);
+
Assertions.assertFalse(dropAfterRename.hasReachedTheTable(getBaseTable(db)));
+ executeSql("ALTER TABLE ivm_base DROP COLUMN v1_renamed");
+
Assertions.assertTrue(dropAfterRename.hasReachedTheTable(getBaseTable(db)),
+ "the column is gone from the table, so the drop has reached
it");
+ Assertions.assertTrue(pendingDrop.hasReachedTheTable(getBaseTable(db)),
+ "and the name that drop asks about is one the table does not
have either");
+
+ // A path that addresses part of a column leaves the column itself in
place, so the question has no
+ // answer: the change is treated as one that has not reached the
table, which keeps the views
+ // invalidated rather than cleared on a look that cannot see the part.
+ DropColumnOp nestedDrop = new
DropColumnOp(ColumnPath.fromDotName("k1.leaf"), null, null);
+
Assertions.assertFalse(nestedDrop.hasReachedTheTable(getBaseTable(db)));
+ }
+
+ /**
+ * Changing a column's type is not judged by the query, and this is the
case that says why: the query
+ * still analyses -- it is projected or not, which is all the check
compares -- while the rows the MV
+ * already holds were computed under the old type. The operation
invalidates on its own then, whether or
+ * not the MV reads the column, which is what keeps a predicate that now
compares something else from
+ * being applied to rows that were filtered under the old meaning.
+ */
+ @Test
+ public void testATypeChangeInvalidatesWithoutAskingTheQuery() throws
Exception {
+ String db = "ivm_type_change";
+ createPartitionedTableWithoutRowBinlog(db);
+ createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n"
+ + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n"
+ + "DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1')\n"
+ + "AS SELECT dt, k1, v1 FROM ivm_base");
+ MTMV mtmv = getMtmv(db);
+ Assertions.assertFalse(mtmv.isIvm());
+
+ executeSql("ALTER TABLE ivm_base ADD COLUMN spare int");
+ executeSql("ALTER TABLE ivm_base MODIFY COLUMN spare bigint");
+
Assertions.assertEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
+ // The change is recorded as the base-table change it is, not as a
query that went away: the query
+ // is still there, and the reason is what tells a reader which of the
two happened.
Assertions.assertFalse(
mtmv.getStatus().getSchemaChangeDetail().contains("no longer
analyzable"),
- "a column the MV does not read leaves the query analyzable,
was: "
- + mtmv.getStatus().getSchemaChangeDetail());
+ "the query is untouched by a type change, was: " +
mtmv.getStatus().getSchemaChangeDetail());
+ }
+
+ /**
+ * A subquery that answers for a name of its own is a dependency only
where the table being changed is
+ * one that subquery reads: what a name falls back to is what the scope
that used to answer for it
+ * holds, so a scope that never read the table holds nothing for it. Two
sibling subqueries are the
+ * shape -- the changed table is one only the first reads, and the `flag`
of the second is its own --
+ * where reading every subquery's own names reads a dependency the change
cannot have made.
+ */
+ @Test
+ public void testASubqueryTheChangeDoesNotReachIsNotADependency() throws
Exception {
+ String db = "ivm_column_change_sibling_scopes";
+ createDatabaseAndUse(db);
+ createTable("CREATE TABLE " + db + ".ivm_base (id int, flag int)
DUPLICATE KEY(id)\n"
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ createTable("CREATE TABLE " + db + ".ivm_other (u int) DUPLICATE
KEY(u)\n"
+ + "DISTRIBUTED BY HASH(u) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ createTable("CREATE TABLE " + db + ".ivm_spare (id int) DUPLICATE
KEY(id)\n"
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n"
+ + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n"
+ + "DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1')\n"
+ + "AS SELECT o.id FROM ivm_base o\n"
+ + "WHERE EXISTS (SELECT 1 FROM ivm_spare c WHERE c.id =
o.id)\n"
+ + " AND EXISTS (SELECT 1 AS flag, COUNT(*) AS n FROM
ivm_other u\n"
+ + " GROUP BY flag HAVING flag = 1)");
+ MTMV mtmv = getMtmv(db);
+
+ long versionBefore = mtmv.getSchemaChangeVersion();
+ executeSql("ALTER TABLE ivm_spare ADD COLUMN flag int default 0");
+ assertNotInvalidated(mtmv, versionBefore,
+ "a name a subquery names itself, where the changed table is
not one that subquery reads,"
+ + " must not invalidate the MV");
+ }
+
+ /**
+ * A subquery inside another one is judged by its own output being read:
an EXISTS is asked whether it
+ * has a row and never what it projects, and an IN around it reads its own
subquery, not the scopes
+ * inside it. Adding the column the inner projection is then read from
leaves every row of the query
+ * where it was, so the MV is left alone -- while a column the IN itself
compares is not.
+ */
+ @Test
+ public void
testAProjectionTheSubqueryAroundItDoesNotReadIsNotADependency() throws
Exception {
+ String db = "ivm_column_change_nested_scope";
+ createDatabaseAndUse(db);
+ createTable("CREATE TABLE " + db + ".ivm_base (id int, k int)
DUPLICATE KEY(id)\n"
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ createTable("CREATE TABLE " + db + ".ivm_mid (k int, id int, spare
int) DUPLICATE KEY(k)\n"
+ + "DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ createTable("CREATE TABLE " + db + ".ivm_spare (id int) DUPLICATE
KEY(id)\n"
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n"
+ + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n"
+ + "DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1')\n"
+ + "AS SELECT o.id FROM ivm_base o\n"
+ + "WHERE o.k IN (SELECT m.k FROM ivm_mid m\n"
+ + " WHERE EXISTS (SELECT spare FROM ivm_spare i WHERE i.id
= m.id))");
+ MTMV mtmv = getMtmv(db);
+
+ long versionBefore = mtmv.getSchemaChangeVersion();
+ executeSql("ALTER TABLE ivm_spare ADD COLUMN spare int default 0");
+ assertNotInvalidated(mtmv, versionBefore,
+ "a projection an EXISTS does not compare, inside a subquery
the IN around it reads the rows"
+ + " of, must not invalidate the MV");
+ }
+
+ /**
+ * A name in a view's own definition has to say which column it was bound
to, wherever it is written.
+ * `QUALIFY` reads a base table's column while that column is there and
whatever else answers for the
+ * name once it is not -- here the view's own `v1`, whose output is a
constant -- which changes the rows
+ * the query returns without changing the columns the view produces. The
judgement re-analyses this
+ * definition, so a name it never bound to a column is one it cannot hold
the change against, and the
+ * view is left holding rows the query no longer returns. Pinned as the
reason it is invalidated with:
+ * the query is gone, because the column it named is.
+ */
+ @Test
+ public void testAQualifyNameIsBoundToItsColumnInTheStoredDefinition()
throws Exception {
+ String db = "ivm_qualify_bound_name";
+ createPartitionedTableWithoutRowBinlog(db);
+ createMvByNereids("CREATE MATERIALIZED VIEW ivm_mv\n"
+ + "BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n"
+ + "DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1')\n"
+ + "AS SELECT 1 AS v1, ROW_NUMBER() OVER (ORDER BY dt) AS rn
FROM ivm_base\n"
+ + "QUALIFY v1 = 1 AND rn = 1");
+ MTMV mtmv = getMtmv(db);
- // A column it does read takes the query away, and that is what the MV
is invalidated with.
executeSql("ALTER TABLE ivm_base DROP COLUMN v1");
Assertions.assertEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
assertUnanalyzableDetail(mtmv);
@@ -1173,6 +1411,28 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
+ "PROPERTIES ('replication_num' = '1', 'binlog.enable' =
'true', 'binlog.format' = 'ROW')");
}
+ /**
+ * The same base table without row binlog. A row binlog table only accepts
the operations marked as safe
+ * for it, and a column rename or a type change is not one of them, so the
changes that need one of those
+ * are pinned against a plain table and a plain MV -- which is where those
two operations reach the
+ * shared base-table hook at all.
+ */
+ private void createPartitionedTableWithoutRowBinlog(String db) throws
Exception {
+ createDatabaseAndUse(db);
+ createTable("CREATE TABLE " + db + ".ivm_base (\n"
+ + " dt date NOT NULL,\n"
+ + " k1 int,\n"
+ + " v1 int\n"
+ + ")\n"
+ + "DUPLICATE KEY(dt, k1)\n"
+ + "PARTITION BY RANGE(dt) (\n"
+ + " PARTITION p202001 VALUES [('2020-01-01'),
('2020-02-01')),\n"
+ + " PARTITION p202002 VALUES [('2020-02-01'),
('2020-03-01'))\n"
+ + ")\n"
+ + "DISTRIBUTED BY HASH(k1) BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1')");
+ }
+
/**
* Gives every MV partition the entry a refresh would have created. A
refresh task aligns before it
* reads a base table, so an MV that has been refreshed has one entry per
partition; the marker tests
@@ -1214,6 +1474,17 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
"the detail must name the reason, was: " +
mtmv.getStatus().getSchemaChangeDetail());
}
+ /**
+ * The MV was left alone by a base-table change. An invalidation moves
three things -- the state, the
+ * version that discards a task result computed against it, and the
snapshot that stops the transparent
+ * rewrite -- so the state alone is only a third of the answer, and the
version is the one of the three
+ * that is observable after a change that names no snapshot.
+ */
+ private void assertNotInvalidated(MTMV mtmv, long versionBefore, String
message) {
+ Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState(), message);
+ Assertions.assertEquals(versionBefore, mtmv.getSchemaChangeVersion(),
message);
+ }
+
private Database getDb(String db) {
return Env.getCurrentInternalCatalog().getDb(db).get();
}
diff --git
a/regression-test/data/mtmv_p0/test_base_drop_col_multi_level_mtmv.out
b/regression-test/data/mtmv_p0/test_base_drop_col_multi_level_mtmv.out
index 2a3165773c1..979eeb3086b 100644
--- a/regression-test/data/mtmv_p0/test_base_drop_col_multi_level_mtmv.out
+++ b/regression-test/data/mtmv_p0/test_base_drop_col_multi_level_mtmv.out
@@ -6,7 +6,7 @@ test_base_drop_col_multi_level_mtmv_mv1 SCHEMA_CHANGE SUCCESS
false
test_base_drop_col_multi_level_mtmv_mv2 NORMAL SUCCESS true
-- !add_col_t1_mv3 --
-test_base_drop_col_multi_level_mtmv_mv3 SCHEMA_CHANGE SUCCESS false
+test_base_drop_col_multi_level_mtmv_mv3 NORMAL SUCCESS true
-- !add_col_t1_mv4 --
test_base_drop_col_multi_level_mtmv_mv4 NORMAL SUCCESS true
diff --git a/regression-test/data/mtmv_p0/test_base_mtmv.out
b/regression-test/data/mtmv_p0/test_base_mtmv.out
index c98bc3167cc..9e54a6372ca 100644
--- a/regression-test/data/mtmv_p0/test_base_mtmv.out
+++ b/regression-test/data/mtmv_p0/test_base_mtmv.out
@@ -9,7 +9,7 @@ multi_mv_test_base_mtmv NORMAL SUCCESS true
multi_mv_test_base_mtmv NORMAL SUCCESS true
-- !rename_column --
-multi_mv_test_base_mtmv SCHEMA_CHANGE SUCCESS false
+multi_mv_test_base_mtmv NORMAL SUCCESS true
-- !success --
multi_mv_test_base_mtmv NORMAL SUCCESS true
@@ -21,7 +21,7 @@ multi_mv_test_base_mtmv SCHEMA_CHANGE SUCCESS false
multi_mv_test_base_mtmv NORMAL SUCCESS true
-- !drop_column --
-multi_mv_test_base_mtmv SCHEMA_CHANGE SUCCESS false
+multi_mv_test_base_mtmv NORMAL SUCCESS true
-- !success --
multi_mv_test_base_mtmv NORMAL SUCCESS true
diff --git
a/regression-test/data/mtmv_p0/test_base_rename_col_multi_level_mtmv.out
b/regression-test/data/mtmv_p0/test_base_rename_col_multi_level_mtmv.out
index d2c39530900..4d64f78d5f6 100644
--- a/regression-test/data/mtmv_p0/test_base_rename_col_multi_level_mtmv.out
+++ b/regression-test/data/mtmv_p0/test_base_rename_col_multi_level_mtmv.out
@@ -6,7 +6,7 @@ test_base_rename_col_multi_level_mtmv_mv1 SCHEMA_CHANGE
SUCCESS false
test_base_rename_col_multi_level_mtmv_mv2 NORMAL SUCCESS true
-- !add_col_t1_mv3 --
-test_base_rename_col_multi_level_mtmv_mv3 SCHEMA_CHANGE SUCCESS false
+test_base_rename_col_multi_level_mtmv_mv3 NORMAL SUCCESS true
-- !add_col_t1_mv4 --
test_base_rename_col_multi_level_mtmv_mv4 NORMAL SUCCESS true
diff --git
a/regression-test/data/mtmv_p0/test_drop_unreferenced_column_mtmv.out
b/regression-test/data/mtmv_p0/test_drop_unreferenced_column_mtmv.out
new file mode 100644
index 00000000000..179b3f06936
--- /dev/null
+++ b/regression-test/data/mtmv_p0/test_drop_unreferenced_column_mtmv.out
@@ -0,0 +1,125 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !mow_baseline --
+1 100
+2 200
+
+-- !mow_state_after_unreferenced_drop --
+test_drop_unreferenced_column_mtmv_mow_mv NORMAL SUCCESS true
+
+-- !mow_state_after_referenced_drop --
+test_drop_unreferenced_column_mtmv_mow_mv SCHEMA_CHANGE SUCCESS false
+
+-- !mow_rows_after_both_drops --
+1 100
+2 200
+
+-- !dup_baseline --
+1 100
+2 200
+
+-- !dup_rows_after_unreferenced_drop --
+1 100
+2 200
+
+-- !scope_drop_baseline --
+1
+
+-- !scope_drop_state --
+test_drop_unreferenced_column_mtmv_scope_drop_mv SCHEMA_CHANGE SUCCESS
false
+
+-- !scope_add_baseline --
+1
+
+-- !scope_add_unreached --
+test_drop_unreferenced_column_mtmv_scope_add_mv NORMAL SUCCESS true
+
+-- !scope_add_state --
+test_drop_unreferenced_column_mtmv_scope_add_mv SCHEMA_CHANGE SUCCESS
false
+
+-- !scope_rename_baseline --
+1
+
+-- !scope_rename_state --
+test_drop_unreferenced_column_mtmv_scope_rename_mv SCHEMA_CHANGE SUCCESS
false
+
+-- !qualified_baseline --
+1
+
+-- !qualified_drop_state --
+test_drop_unreferenced_column_mtmv_qual_mv NORMAL SUCCESS true
+
+-- !qualified_add_state --
+test_drop_unreferenced_column_mtmv_qual_mv NORMAL SUCCESS true
+
+-- !qualified_rows --
+1
+
+-- !job_add_baseline --
+test_drop_unreferenced_column_mtmv_job_add_mv NORMAL SUCCESS true
+
+-- !job_add_cols_baseline --
+test_drop_unreferenced_column_mtmv_job_add_cols_mv NORMAL SUCCESS true
+
+-- !projection_baseline --
+1
+
+-- !projection_state --
+test_drop_unreferenced_column_mtmv_proj_mv SCHEMA_CHANGE SUCCESS false
+
+-- !alias_baseline --
+0
+
+-- !alias_state --
+test_drop_unreferenced_column_mtmv_alias_mv SCHEMA_CHANGE SUCCESS false
+
+-- !ignored_projection_state --
+test_drop_unreferenced_column_mtmv_ignored_mv NORMAL SUCCESS true
+
+-- !ignored_projection_rows --
+1
+
+-- !named_not_read_state --
+test_drop_unreferenced_column_mtmv_named_mv NORMAL SUCCESS true
+
+-- !sibling_baseline --
+1
+
+-- !sibling_state --
+test_drop_unreferenced_column_mtmv_sibling_mv NORMAL SUCCESS true
+
+-- !sibling_rows --
+1
+
+-- !nested_baseline --
+1
+
+-- !nested_state --
+test_drop_unreferenced_column_mtmv_nested_mv NORMAL SUCCESS true
+
+-- !nested_rows --
+1
+
+-- !qualify_baseline --
+0
+
+-- !qualify_state --
+test_drop_unreferenced_column_mtmv_qualify_mv SCHEMA_CHANGE SUCCESS false
+
+-- !alias_of_another_column_baseline --
+1
+
+-- !alias_of_another_column_state --
+test_drop_unreferenced_column_mtmv_alias_of_mv NORMAL SUCCESS true
+
+-- !alias_of_another_column_rows --
+1
+
+-- !alias_in_subquery_baseline --
+1
+
+-- !alias_in_subquery_state --
+test_drop_unreferenced_column_mtmv_alias_in_mv NORMAL SUCCESS true
+
+-- !alias_in_subquery_rows --
+1
+
diff --git
a/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy
b/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy
index 1bb6cb99c26..c3df98913ae 100644
---
a/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy
+++
b/regression-test/suites/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.groovy
@@ -109,11 +109,11 @@ suite("test_ivm_drop_referenced_column_baseline_rebuild")
{
order_qt_mv_rows_baseline "SELECT grp, cnt, total FROM ${mvName}"
// ------------------------------------- 2. unreferenced column: no
baseline invalidation
- // The IVM baseline itself is untouched -- no partition requirement is
raised. The shared base-table
- // change hook still moves the MV into SCHEMA_CHANGE for a column change,
though, and that state is
- // what the refresh below reads: it is escalated to a whole-MV COMPLETE.
Narrowing the hook so a
- // change that re-analyses cleanly leaves an IVM MV alone is PR 4's S1-5;
pinned here so the
- // escalation cannot pass unnoticed until then.
+ // Neither half is touched: no partition requirement is raised, and the
shared base-table change hook
+ // leaves the MV alone as well -- its criterion is a re-analysis of the
MV's own query, and a column
+ // the query does not name leaves that query analysable, so there is no
rebuild for the state to stand
+ // for. How that shows up is what the refresh below reports: it stays on
the incremental path, which is
+ // the path that records no refresh mode at all, instead of being
escalated to a whole-MV COMPLETE.
def before = ddlJobCount(tableName)
sql """ALTER TABLE ${tableName} DROP COLUMN spare"""
waitDdlFinished(tableName, before)
@@ -121,11 +121,13 @@ suite("test_ivm_drop_referenced_column_baseline_rebuild")
{
sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL"""
task = waitTerminalTask(mvName)
assertEquals("SUCCESS", task.Status.toString(),
- "dropping an unreferenced column must not invalidate the IVM
baseline: " + task.ErrorMsg)
- assertEquals("COMPLETE", task.RefreshMode.toString(),
- "the shared hook still moves the MV into SCHEMA_CHANGE, so this
refresh is escalated")
- assertEquals("1", task.IvmRebuiltPartitions.toString(),
- "and the escalated refresh reports the partition it rebuilt
instead of the request it got")
+ "dropping an unreferenced column must not invalidate the MV: " +
task.ErrorMsg)
+ // Only the partition and whole-MV attempts set a mode; an incremental one
leaves the column unset,
+ // which comes back as the same literal the folding in the other suites
turns into a token.
+ assertEquals("\\N", task.RefreshMode.toString(),
+ "the refresh must not be escalated to a whole-MV COMPLETE, so it
records no refresh mode")
+ assertEquals("0", task.IvmRebuiltPartitions.toString(),
+ "and it rebuilds no partition the request did not ask for")
// --------------------------- 3. referenced column: the refresh can no
longer analyse
before = ddlJobCount(tableName)
diff --git
a/regression-test/suites/mtmv_p0/test_base_drop_col_multi_level_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_base_drop_col_multi_level_mtmv.groovy
index 14cb8a2faa4..3a2f8c83dac 100644
--- a/regression-test/suites/mtmv_p0/test_base_drop_col_multi_level_mtmv.groovy
+++ b/regression-test/suites/mtmv_p0/test_base_drop_col_multi_level_mtmv.groovy
@@ -136,6 +136,8 @@ suite("test_base_drop_col_multi_level_mtmv","mtmv") {
mv_rewrite_success_without_check_chosen(querySql, mvName2)
mv_not_part_in(querySql, mvName1)
- mv_not_part_in(querySql, mvName3)
+ // mv3 is the query being asked, and the column this change took away is
not one it names: the MV it
+ // stands for is untouched, so it stays a candidate -- and it is the one
the plan picks.
+ mv_rewrite_success(querySql, mvName3)
mv_rewrite_fail(querySql, mvName4)
}
diff --git
a/regression-test/suites/mtmv_p0/test_base_rename_col_multi_level_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_base_rename_col_multi_level_mtmv.groovy
index f1ad7a59b44..ee89542ef87 100644
---
a/regression-test/suites/mtmv_p0/test_base_rename_col_multi_level_mtmv.groovy
+++
b/regression-test/suites/mtmv_p0/test_base_rename_col_multi_level_mtmv.groovy
@@ -135,6 +135,8 @@ suite("test_base_rename_col_multi_level_mtmv","mtmv") {
mv_rewrite_success_without_check_chosen(querySql, mvName2)
mv_not_part_in(querySql, mvName1)
- mv_not_part_in(querySql, mvName3)
+ // mv3 is the query being asked, and the column this change took away is
not one it names: the MV it
+ // stands for is untouched, so it stays a candidate -- and it is the one
the plan picks.
+ mv_rewrite_success(querySql, mvName3)
mv_rewrite_fail(querySql, mvName4)
}
diff --git
a/regression-test/suites/mtmv_p0/test_drop_unreferenced_column_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_drop_unreferenced_column_mtmv.groovy
new file mode 100644
index 00000000000..28a2afa9969
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_drop_unreferenced_column_mtmv.groovy
@@ -0,0 +1,659 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.junit.Assert;
+
+/**
+ * Which base-table column changes invalidate a materialized view.
+ *
+ * <p>A dropped column is judged by re-analysing the MV's own query -- but
only where that question has an
+ * answer. The query is analysed against the table as it is when the alter
reaches the MV hook, and unless
+ * the change is a light one it has not been applied yet at that point: it was
submitted as a job, the
+ * column is still there, and every query still analyses. An invalidation
decided on that answer would be
+ * about the table from before the change, which is exactly where a dropped
and re-added column leaves the
+ * ABA the invalidation exists for. So a change that is not in place keeps
invalidating, and the two halves
+ * of this suite pin the two answers:
+ * <ol>
+ * <li>a merge-on-write table, where dropping a value column is a light
change: the MV whose query does
+ * not name the column is left alone, and the MV whose query names it is
invalidated;</li>
+ * <li>a duplicate table, where the same drop is not a light one and a job
does the data rewrite: the
+ * column is out of the table's schema before the hook runs all the
same, so the same answer holds.
+ * What the gate is for is the other side of that -- a change a job has
not applied yet, where the
+ * query would be analysed against the table from before it. There the
MV is invalidated, which is
+ * what every column change did before the queries were asked at
all.</li>
+ * </ol>
+ *
+ * <p>That second half is the integrated cluster's. Cloud has no table a job
applies a column change to:
+ * `light_schema_change` there is the platform's rather than the table's -- a
table created with it false
+ * reports and uses it true, and the change lands in the statement -- so on
cloud the change is judged like
+ * any light one and the answer is the first half's. Both answers are asserted
where they differ, and the
+ * reason is written where the shapes are.
+ *
+ * <p>The rest of the suite is about which column a name in the query answers
for, because that is what
+ * makes a change to a column nothing reads. A name is a column of the table
the change is about only where
+ * the query reads it from there, and it is the scopes' to answer for where
the query resolves it across a
+ * scope boundary: what is left is a name bound to another table inside the
query's own scope, which no
+ * later change can move.
+ *
+ * <p>The second half is also where the consequence is observable: the rewrite
reaches the MV on that table
+ * and not on a merge-on-write one, so the state the change records is what is
left to report on the first.
+ * The IVM side of the same change, where the state is what escalates the next
refresh to a whole-MV
+ * COMPLETE, is pinned in the ivm directory.
+ */
+suite("test_drop_unreferenced_column_mtmv") {
+ String dbName = context.config.getDbNameByFile(context.file)
+ String suiteName = "test_drop_unreferenced_column_mtmv"
+
+ // The state a view is left in, kept out of the goldens where the answer
itself is not the same on both
+ // kinds of cluster: a golden holds one answer, and the drop a job would
have applied (below) is not a
+ // drop cloud has.
+ def mvStateOf = { String name ->
+ (sql("select State from mv_infos('database'='${dbName}') where
Name='${name}'")[0][0]) as String
+ }
+
+ // ------------------------------------------------- 1. the change is in
place: the query decides
+ String mowTable = "${suiteName}_mow_table"
+ String mowMv = "${suiteName}_mow_mv"
+
+ sql """drop materialized view if exists ${mowMv}"""
+ sql """drop table if exists ${mowTable}"""
+ sql """
+ CREATE TABLE ${mowTable}
+ (
+ k1 INT NOT NULL,
+ amount BIGINT,
+ spare BIGINT
+ )
+ UNIQUE KEY(k1)
+ DISTRIBUTED BY HASH(k1) BUCKETS 2
+ PROPERTIES ("replication_num" = "1",
"enable_unique_key_merge_on_write" = "true")
+ """
+ sql """INSERT INTO ${mowTable} VALUES (1, 100, 7), (2, 200, 8)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${mowMv}
+ BUILD DEFERRED REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(k1) BUCKETS 2
+ PROPERTIES ("replication_num" = "1")
+ AS SELECT k1, SUM(amount) AS total FROM ${mowTable} GROUP BY k1
+ """
+ sql """REFRESH MATERIALIZED VIEW ${mowMv} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(mowMv)
+ order_qt_mow_baseline "SELECT k1, total FROM ${mowMv}"
+
+ // A column the query does not name. Dropping it gives this MV nothing to
recompute, so it is not
+ // invalidated: it stays a refresh candidate, and the state is where that
shows.
+ sql """ALTER TABLE ${mowTable} DROP COLUMN spare"""
+ assertEquals("FINISHED", getAlterColumnFinalState("${mowTable}"))
+ order_qt_mow_state_after_unreferenced_drop "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${mowMv}'"
+
+ // A column the query names. The MV cannot be computed from the table any
more, so it is invalidated.
+ sql """ALTER TABLE ${mowTable} DROP COLUMN amount"""
+ assertEquals("FINISHED", getAlterColumnFinalState("${mowTable}"))
+ order_qt_mow_state_after_referenced_drop "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${mowMv}'"
+ // Neither column was one the rows the MV holds depended on: the
invalidation above is not about the
+ // data being wrong, it is about the query the data stands for.
+ order_qt_mow_rows_after_both_drops "SELECT k1, total FROM ${mowMv}"
+
+ // ------------------------- 2. the same drop on a table whose data a job
rewrites: same answer
+ String dupTable = "${suiteName}_dup_table"
+ String dupMv = "${suiteName}_dup_mv"
+ String dupQuery = "SELECT k1, SUM(amount) AS total FROM ${dupTable} GROUP
BY k1"
+
+ String scopeDropOuter = "${suiteName}_scope_drop_outer"
+ String scopeDropInner = "${suiteName}_scope_drop_inner"
+ String scopeDropMv = "${suiteName}_scope_drop_mv"
+ String scopeAddOuter = "${suiteName}_scope_add_outer"
+ String scopeAddInner = "${suiteName}_scope_add_inner"
+ String scopeAddMv = "${suiteName}_scope_add_mv"
+
+ sql """drop materialized view if exists ${dupMv}"""
+ sql """drop table if exists ${dupTable}"""
+ sql """
+ CREATE TABLE ${dupTable}
+ (
+ k1 INT,
+ amount BIGINT,
+ spare BIGINT
+ )
+ DUPLICATE KEY(k1)
+ DISTRIBUTED BY HASH(k1) BUCKETS 2
+ PROPERTIES ("replication_num" = "1", "light_schema_change" = "false")
+ """
+ sql """INSERT INTO ${dupTable} VALUES (1, 100, 7), (2, 200, 8)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${dupMv}
+ BUILD DEFERRED REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(k1) BUCKETS 2
+ PROPERTIES ("replication_num" = "1")
+ AS SELECT k1, SUM(amount) AS total FROM ${dupTable} GROUP BY k1
+ """
+ sql """REFRESH MATERIALIZED VIEW ${dupMv} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(dupMv)
+ order_qt_dup_baseline "SELECT k1, total FROM ${dupMv}"
+ // This MV is reachable by the rewrite, which is what makes the answer
below observable: the MV takes
+ // part in it before the change, and a change that invalidated the MV
would take it out.
+ mv_rewrite_success_without_check_chosen(dupQuery, dupMv)
+
+ // The other half of the answer, on a table where a job applies the change
rather than the statement.
+ // The hook runs where that job has not run yet, so the table still holds
the column and every query
+ // analyses against it: nothing can be concluded about the column from a
query, and the MV is
+ // invalidated the way it was before the queries were asked at all.
+ //
+ // Cloud has no such table. There `light_schema_change` is not the table's
to choose: a table created
+ // with it false reports it true and takes the column change in the
statement itself (a schema change
+ // there is a metadata change with no data rewrite behind it), so this
drop is the light one of the half
+ // above and gets that half's answer -- the change reached the table, the
query was asked, and the MV is
+ // left where it is. Both halves of the answer are pinned here: the state
by an assertion, because a
+ // golden can only hold one of the two.
+ sql """ALTER TABLE ${dupTable} DROP COLUMN spare"""
+ assertEquals("FINISHED", getAlterColumnFinalState("${dupTable}"))
+ assertEquals(isCloudMode() ? "NORMAL" : "SCHEMA_CHANGE", mvStateOf(dupMv))
+ if (isCloudMode()) {
+ // The change does not reach this MV, so it is still a candidate for
the rewrite.
+ mv_rewrite_success_without_check_chosen(dupQuery, dupMv)
+ } else {
+ mv_not_part_in(dupQuery, dupMv)
+ }
+ order_qt_dup_rows_after_unreferenced_drop "SELECT k1, total FROM ${dupMv}"
+
+ // ---- the name a query reaches a column by can move, and that is the
query's to judge ----
+ // `flag` below is unqualified inside the subquery, so it is the inner
table's column while that table
+ // has one: the column nearest to a name in the query's scopes answers for
it. Taking that column away
+ // leaves the name to the outer table and the query still analyses with
the columns it always produced,
+ // while its rows are the ones of the column that went away.
+ sql """drop materialized view if exists ${scopeDropMv}"""
+ sql """drop table if exists ${scopeDropOuter}"""
+ sql """drop table if exists ${scopeDropInner}"""
+ sql """
+ CREATE TABLE ${scopeDropOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${scopeDropInner} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${scopeDropOuter} VALUES (1, 0)"""
+ sql """INSERT INTO ${scopeDropInner} VALUES (1, 1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${scopeDropMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${scopeDropOuter} o
+ WHERE EXISTS (SELECT 1 FROM ${scopeDropInner} i WHERE i.id = o.id AND
flag = 1)
+ """
+ waitingMTMVTaskFinishedByMvName(scopeDropMv)
+ order_qt_scope_drop_baseline "SELECT id FROM ${scopeDropMv}"
+ sql """ALTER TABLE ${scopeDropInner} DROP COLUMN flag"""
+ order_qt_scope_drop_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${scopeDropMv}'"
+
+ // And the same name can be taken over by a column that arrives. The inner
table has no `flag` here, so
+ // the name is the outer table's to answer; a column added inside the
subquery's scope answers for it
+ // from then on. A column added that no query reaches a name by leaves
every MV alone.
+ sql """drop materialized view if exists ${scopeAddMv}"""
+ sql """drop table if exists ${scopeAddOuter}"""
+ sql """drop table if exists ${scopeAddInner}"""
+ sql """
+ CREATE TABLE ${scopeAddOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${scopeAddInner} (id INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${scopeAddOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${scopeAddInner} VALUES (1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${scopeAddMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${scopeAddOuter} o
+ WHERE EXISTS (SELECT 1 FROM ${scopeAddInner} i WHERE i.id = o.id AND
flag = 1)
+ """
+ waitingMTMVTaskFinishedByMvName(scopeAddMv)
+ order_qt_scope_add_baseline "SELECT id FROM ${scopeAddMv}"
+ sql """ALTER TABLE ${scopeAddInner} ADD COLUMN unrelated BIGINT"""
+ order_qt_scope_add_unreached "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${scopeAddMv}'"
+ sql """ALTER TABLE ${scopeAddInner} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_scope_add_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${scopeAddMv}'"
+
+ // A rename takes a name over the same way an add gives one, and it does
it with two names at once: the
+ // one the query spelled, which the column no longer answers for, and the
one that answers for it from
+ // then on. Neither of the two says on its own whether the query moved --
the old name is one no query
+ // reaches any more, and the new one is only a move where the old name was
the query's before -- so the
+ // rename reports both. Here the inner table has `x` and no `flag`, so
`flag` is the outer table's, and
+ // renaming `x` to `flag` hands the name to the inner scope: the query
goes on producing `o.id` out of
+ // the rows of a column it never read, and the MV is invalidated.
+ String scopeRenameOuter = "${suiteName}_scope_rename_outer"
+ String scopeRenameInner = "${suiteName}_scope_rename_inner"
+ String scopeRenameMv = "${suiteName}_scope_rename_mv"
+ sql """drop materialized view if exists ${scopeRenameMv}"""
+ sql """drop table if exists ${scopeRenameOuter}"""
+ sql """drop table if exists ${scopeRenameInner}"""
+ sql """
+ CREATE TABLE ${scopeRenameOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${scopeRenameInner} (id INT, x INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${scopeRenameOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${scopeRenameInner} VALUES (1, 0)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${scopeRenameMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${scopeRenameOuter} o
+ WHERE EXISTS (SELECT 1 FROM ${scopeRenameInner} i WHERE i.id = o.id
AND flag = 1)
+ """
+ waitingMTMVTaskFinishedByMvName(scopeRenameMv)
+ order_qt_scope_rename_baseline "SELECT id FROM ${scopeRenameMv}"
+ sql """ALTER TABLE ${scopeRenameInner} RENAME COLUMN x flag"""
+ order_qt_scope_rename_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${scopeRenameMv}'"
+
+ // ---- a name another table answers for is not this one's to give or take
----
+ // `flag` here is written with the qualifier of the second table, so what
the query reads does not turn
+ // on anything the first table does with a `flag` of its own: a name bound
where it is written is one no
+ // later change can move. The first table's `flag` is 0 and the second's
is 1, so the row the MV holds is
+ // the one the qualified binding produces -- a query that had read the
first table's column would hold
+ // none -- and taking that column away, or giving it back, leaves this
query and this MV alone.
+ String qualOuter = "${suiteName}_qual_outer"
+ String qualInner = "${suiteName}_qual_inner"
+ String qualMv = "${suiteName}_qual_mv"
+
+ sql """drop materialized view if exists ${qualMv}"""
+ sql """drop table if exists ${qualOuter}"""
+ sql """drop table if exists ${qualInner}"""
+ sql """
+ CREATE TABLE ${qualOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${qualInner} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${qualOuter} VALUES (1, 0)"""
+ sql """INSERT INTO ${qualInner} VALUES (1, 1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${qualMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT a.id FROM ${qualOuter} a JOIN ${qualInner} b ON a.id = b.id
WHERE b.flag = 1
+ """
+ waitingMTMVTaskFinishedByMvName(qualMv)
+ order_qt_qualified_baseline "SELECT id FROM ${qualMv}"
+ sql """ALTER TABLE ${qualOuter} DROP COLUMN flag"""
+ order_qt_qualified_drop_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${qualMv}'"
+ sql """ALTER TABLE ${qualOuter} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_qualified_add_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${qualMv}'"
+ order_qt_qualified_rows "SELECT id FROM ${qualMv}"
+
+ // ---- an add a job applies: there is no query to ask ----
+ // The hook of a change that is not a light one runs before the job has
applied it, so the table is the one
+ // from before the change and every query is analysed against that. A
column that is not in the table yet
+ // is one no query can be looked at for, and the MV is invalidated -- for
a column added on its own and for
+ // a batch of them, which is how an add is written either way. What the
add would do once the job lands is
+ // not readable here, and a name it may answer for is one the rows of
these views were not computed under.
+ String jobAddTable = "${suiteName}_job_add_table"
+ String jobAddMv = "${suiteName}_job_add_mv"
+ String jobAddColsTable = "${suiteName}_job_add_cols_table"
+ String jobAddColsMv = "${suiteName}_job_add_cols_mv"
+ sql """drop materialized view if exists ${jobAddMv}"""
+ sql """drop materialized view if exists ${jobAddColsMv}"""
+ sql """drop table if exists ${jobAddTable}"""
+ sql """drop table if exists ${jobAddColsTable}"""
+ sql """
+ CREATE TABLE ${jobAddTable} (id INT, amount INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1", "light_schema_change" = "false")
+ """
+ sql """
+ CREATE TABLE ${jobAddColsTable} (id INT, amount INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1", "light_schema_change" = "false")
+ """
+ sql """INSERT INTO ${jobAddTable} VALUES (1, 100)"""
+ sql """INSERT INTO ${jobAddColsTable} VALUES (1, 100)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${jobAddMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT id, SUM(amount) AS total FROM ${jobAddTable} GROUP BY id
+ """
+ sql """
+ CREATE MATERIALIZED VIEW ${jobAddColsMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT id, SUM(amount) AS total FROM ${jobAddColsTable} GROUP BY id
+ """
+ waitingMTMVTaskFinishedByMvName(jobAddMv)
+ waitingMTMVTaskFinishedByMvName(jobAddColsMv)
+ order_qt_job_add_baseline "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${jobAddMv}'"
+ order_qt_job_add_cols_baseline "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${jobAddColsMv}'"
+ // On cloud the same add is a light one -- `light_schema_change` is not
the table's to choose there, as
+ // the drop above writes down -- so the column is in the table when the
hook runs and the query is asked:
+ // neither view reads it, and both are left where they are.
+ sql """ALTER TABLE ${jobAddTable} ADD COLUMN spare INT"""
+ assertEquals(isCloudMode() ? "NORMAL" : "SCHEMA_CHANGE",
mvStateOf(jobAddMv))
+ sql """ALTER TABLE ${jobAddColsTable} ADD COLUMN (spare INT, other INT)"""
+ assertEquals(isCloudMode() ? "NORMAL" : "SCHEMA_CHANGE",
mvStateOf(jobAddColsMv))
+
+ // ---- a name a subquery's own projection answers for ----
+ // The subquery below projects `flag` while the inner table has no column
of that name, so the name is
+ // the outer table's to answer for and the subquery projects the outer
column: the MV holds the row.
+ // Giving the inner table a column of that name hands the name to it, so
the projection reads the inner
+ // column from then on and the query returns no row -- while every column
the MV produces is the one it
+ // always produced. Nothing is read here that the rows of the view are
computed from, and the name is
+ // still the change's to answer for: a column of a subquery's own output
is where a name can move
+ // without moving an output of the view.
+ String projOuter = "${suiteName}_proj_outer"
+ String projInner = "${suiteName}_proj_inner"
+ String projMv = "${suiteName}_proj_mv"
+ sql """drop materialized view if exists ${projMv}"""
+ sql """drop table if exists ${projOuter}"""
+ sql """drop table if exists ${projInner}"""
+ sql """
+ CREATE TABLE ${projOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${projInner} (x INT) DUPLICATE KEY(x)
+ DISTRIBUTED BY HASH(x) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${projOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${projInner} VALUES (1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${projMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${projOuter} o WHERE o.flag IN (SELECT flag FROM
${projInner} i)
+ """
+ waitingMTMVTaskFinishedByMvName(projMv)
+ order_qt_projection_baseline "SELECT id FROM ${projMv}"
+ sql """ALTER TABLE ${projInner} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_projection_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${projMv}'"
+
+ // ---- a name a subquery names itself ----
+ // `flag` here is written unqualified inside the subquery and the subquery
names a `flag` of its own, so
+ // the name is the table's column while the table has one and the
subquery's own value once it does not:
+ // the group by and the having then read the constant, the subquery has a
row, and the query returns a
+ // row the view's rows do not have. Nothing of the view's is computed from
a column of the changed table
+ // afterwards, so the check has to read what the subquery itself names.
+ String aliasOuter = "${suiteName}_alias_outer"
+ String aliasInner = "${suiteName}_alias_inner"
+ String aliasMv = "${suiteName}_alias_mv"
+ sql """drop materialized view if exists ${aliasMv}"""
+ sql """drop table if exists ${aliasOuter}"""
+ sql """drop table if exists ${aliasInner}"""
+ sql """
+ CREATE TABLE ${aliasOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ """
+ sql """
+ CREATE TABLE ${aliasInner} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ """
+ sql """INSERT INTO ${aliasOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${aliasInner} VALUES (1, 0)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${aliasMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ AS SELECT o.id FROM ${aliasOuter} o WHERE EXISTS (
+ SELECT 1 AS flag, COUNT(*) AS n FROM ${aliasInner} i
+ WHERE i.id = o.id GROUP BY flag HAVING flag = 1)
+ """
+ waitingMTMVTaskFinishedByMvName(aliasMv)
+ order_qt_alias_baseline "SELECT COUNT(*) FROM ${aliasMv}"
+ sql """ALTER TABLE ${aliasInner} DROP COLUMN flag"""
+ order_qt_alias_state "select Name,State,RefreshState,SyncWithBaseTables
from mv_infos('database'='${dbName}') where Name='${aliasMv}'"
+
+ // ---- a name a subquery projects and never compares ----
+ // The subquery below projects `spare`, which is only the outer table's
while the outer table is the one
+ // that has it; an EXISTS tests the rows of its subquery and not what it
projects, so the same rows come
+ // back after the inner table is given a column of that name. Reading the
subquery's scope has to leave
+ // that case alone, or a change this view does not depend on takes its
rewrite snapshot with it.
+ String ignoredOuter = "${suiteName}_ignored_outer"
+ String ignoredInner = "${suiteName}_ignored_inner"
+ String ignoredMv = "${suiteName}_ignored_mv"
+ sql """drop materialized view if exists ${ignoredMv}"""
+ sql """drop table if exists ${ignoredOuter}"""
+ sql """drop table if exists ${ignoredInner}"""
+ sql """
+ CREATE TABLE ${ignoredOuter} (id INT, spare INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ """
+ sql """
+ CREATE TABLE ${ignoredInner} (id INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ """
+ sql """INSERT INTO ${ignoredOuter} VALUES (1, 7)"""
+ sql """INSERT INTO ${ignoredInner} VALUES (1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${ignoredMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ AS SELECT o.id FROM ${ignoredOuter} o
+ WHERE EXISTS (SELECT spare FROM ${ignoredInner} i WHERE i.id = o.id)
+ """
+ waitingMTMVTaskFinishedByMvName(ignoredMv)
+ sql """ALTER TABLE ${ignoredInner} ADD COLUMN spare INT DEFAULT 0"""
+ order_qt_ignored_projection_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${ignoredMv}'"
+ order_qt_ignored_projection_rows "SELECT id FROM ${ignoredMv}"
+
+ // ---- a name a subquery names and never reads ----
+ // The subquery below names a `flag` of its own and no reference in it
resolves to that name: the compared
+ // `flag` is the outer table's, written with its qualifier, and giving the
inner table a column of that
+ // name changes the subquery's own projection and nothing the query reads.
Naming a value is not reading
+ // it, so this MV is left where it is.
+ String namedOuter = "${suiteName}_named_outer"
+ String namedInner = "${suiteName}_named_inner"
+ String namedMv = "${suiteName}_named_mv"
+ sql """drop materialized view if exists ${namedMv}"""
+ sql """drop table if exists ${namedOuter}"""
+ sql """drop table if exists ${namedInner}"""
+ sql """
+ CREATE TABLE ${namedOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ """
+ sql """
+ CREATE TABLE ${namedInner} (id INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ """
+ sql """INSERT INTO ${namedOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${namedInner} VALUES (1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${namedMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" =
\"1\")
+ AS SELECT o.id FROM ${namedOuter} o WHERE o.flag IN (SELECT 1 AS flag
FROM ${namedInner} i)
+ """
+ waitingMTMVTaskFinishedByMvName(namedMv)
+ sql """ALTER TABLE ${namedInner} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_named_not_read_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${namedMv}'"
+
+ // ---- a name a subquery names itself, in a scope the change reaches
nothing of ----
+ // The name below is the second subquery's own, and the table being
changed is one only the first
+ // subquery reads: nothing inside the second scope ever answered for the
changed column, so the name it
+ // answers for is not one the change moved, and the MV is left where it
is. Reading what a subquery
+ // names itself has to be limited to the scopes the changed table is one
of, or a scope elsewhere in the
+ // query that happens to name the same word is read as a dependency of
this view's rows.
+ String siblingOuter = "${suiteName}_sibling_outer"
+ String siblingOther = "${suiteName}_sibling_other"
+ String siblingChanged = "${suiteName}_sibling_changed"
+ String siblingMv = "${suiteName}_sibling_mv"
+ sql """drop materialized view if exists ${siblingMv}"""
+ sql """drop table if exists ${siblingOuter}"""
+ sql """drop table if exists ${siblingOther}"""
+ sql """drop table if exists ${siblingChanged}"""
+ sql """
+ CREATE TABLE ${siblingOuter} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${siblingOther} (u INT) DUPLICATE KEY(u)
+ DISTRIBUTED BY HASH(u) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${siblingChanged} (id INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${siblingOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${siblingOther} VALUES (1)"""
+ sql """INSERT INTO ${siblingChanged} VALUES (1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${siblingMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${siblingOuter} o
+ WHERE EXISTS (SELECT 1 FROM ${siblingChanged} c WHERE c.id = o.id)
+ AND EXISTS (SELECT 1 AS flag, COUNT(*) AS n FROM ${siblingOther} u
GROUP BY flag HAVING flag = 1)
+ """
+ waitingMTMVTaskFinishedByMvName(siblingMv)
+ order_qt_sibling_baseline "SELECT id FROM ${siblingMv}"
+ sql """ALTER TABLE ${siblingChanged} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_sibling_state "select Name,State,RefreshState,SyncWithBaseTables
from mv_infos('database'='${dbName}') where Name='${siblingMv}'"
+ order_qt_sibling_rows "SELECT id FROM ${siblingMv}"
+
+ // ---- a projection of a subquery that the subquery around it never reads
----
+ // The IN compares the rows of its own subquery; the EXISTS inside that
subquery is asked only whether
+ // it has a row, never what it projects. So the `spare` the inner subquery
projects is the inner table's
+ // column only after the add, and either way the rows of this query are
the same -- the scope of the
+ // inner EXISTS is not read by the IN around it, and is judged with the
EXISTS it belongs to.
+ String nestedOuter = "${suiteName}_nested_outer"
+ String nestedMid = "${suiteName}_nested_mid"
+ String nestedInner = "${suiteName}_nested_inner"
+ String nestedMv = "${suiteName}_nested_mv"
+ sql """drop materialized view if exists ${nestedMv}"""
+ sql """drop table if exists ${nestedOuter}"""
+ sql """drop table if exists ${nestedMid}"""
+ sql """drop table if exists ${nestedInner}"""
+ sql """
+ CREATE TABLE ${nestedOuter} (id INT, k INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${nestedMid} (k INT, id INT, spare INT) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${nestedInner} (id INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${nestedOuter} VALUES (1, 1)"""
+ sql """INSERT INTO ${nestedMid} VALUES (1, 1, 7)"""
+ sql """INSERT INTO ${nestedInner} VALUES (1)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${nestedMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${nestedOuter} o
+ WHERE o.k IN (SELECT m.k FROM ${nestedMid} m
+ WHERE EXISTS (SELECT spare FROM ${nestedInner} i WHERE i.id =
m.id))
+ """
+ waitingMTMVTaskFinishedByMvName(nestedMv)
+ order_qt_nested_baseline "SELECT id FROM ${nestedMv}"
+ sql """ALTER TABLE ${nestedInner} ADD COLUMN spare INT DEFAULT 0"""
+ order_qt_nested_state "select Name,State,RefreshState,SyncWithBaseTables
from mv_infos('database'='${dbName}') where Name='${nestedMv}'"
+ order_qt_nested_rows "SELECT id FROM ${nestedMv}"
+
+ // ---- a name written where the definition does not say which column it
was ----
+ // The judgement re-analyses the view's own definition, so a name in it
that the definition never bound
+ // to a column is one the re-analysis can bind elsewhere: `QUALIFY` below
reads the base table's `flag`
+ // while it is there, and the alias once it is not, which changes the rows
the query returns without
+ // changing the columns the view produces. A view's definition has to
carry the column each of its names
+ // was bound to, wherever that name is written, or a change to that column
is one this cannot see.
+ String qualifyTable = "${suiteName}_qualify_table"
+ String qualifyMv = "${suiteName}_qualify_mv"
+ sql """drop materialized view if exists ${qualifyMv}"""
+ sql """drop table if exists ${qualifyTable}"""
+ sql """
+ CREATE TABLE ${qualifyTable} (id INT, flag INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${qualifyTable} VALUES (1, 0)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${qualifyMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(flag) BUCKETS 1 PROPERTIES ("replication_num" =
"1")
+ AS SELECT 1 AS flag, ROW_NUMBER() OVER (ORDER BY id) AS rn FROM
${qualifyTable}
+ QUALIFY flag = 1 AND rn = 1
+ """
+ waitingMTMVTaskFinishedByMvName(qualifyMv)
+ order_qt_qualify_baseline "SELECT COUNT(*) FROM ${qualifyMv}"
+ sql """ALTER TABLE ${qualifyTable} DROP COLUMN flag"""
+ order_qt_qualify_state "select Name,State,RefreshState,SyncWithBaseTables
from mv_infos('database'='${dbName}') where Name='${qualifyMv}'"
+
+ // ---- a name an alias gave to a column of another name ----
+ // The projection below is called `flag` and reads the column `x`: an
alias renames what a slot is known
+ // by without moving the column it comes from, so a change to a column
named `flag` is a change to
+ // something this view's rows are not computed from -- the column it reads
keeps its own name. Holding
+ // the change against the name the slot is called invalidated the view for
the name it answers to
+ // rather than the column it reads.
+ String aliasOfAColumnTable = "${suiteName}_alias_of_table"
+ String aliasOfAColumnMv = "${suiteName}_alias_of_mv"
+ sql """drop materialized view if exists ${aliasOfAColumnMv}"""
+ sql """drop table if exists ${aliasOfAColumnTable}"""
+ sql """
+ CREATE TABLE ${aliasOfAColumnTable} (id INT, x INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${aliasOfAColumnTable} VALUES (1, 5)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${aliasOfAColumnMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT d.id FROM (SELECT c.id, c.x AS flag FROM
${aliasOfAColumnTable} c) d WHERE d.flag = 5
+ """
+ waitingMTMVTaskFinishedByMvName(aliasOfAColumnMv)
+ order_qt_alias_of_another_column_baseline "SELECT id FROM
${aliasOfAColumnMv}"
+ sql """ALTER TABLE ${aliasOfAColumnTable} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_alias_of_another_column_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${aliasOfAColumnMv}'"
+ order_qt_alias_of_another_column_rows "SELECT id FROM ${aliasOfAColumnMv}"
+
+ // ---- the same, read from the subquery a comparison takes its rows from
----
+ // `d.flag` below is the alias over `c.x` again, read by the IN rather
than by a filter. The same
+ // question is asked of it -- which column does this name read -- and the
same name is not the change's.
+ String aliasInTable = "${suiteName}_alias_in_table"
+ String aliasInOuter = "${suiteName}_alias_in_outer"
+ String aliasInMv = "${suiteName}_alias_in_mv"
+ sql """drop materialized view if exists ${aliasInMv}"""
+ sql """drop table if exists ${aliasInTable}"""
+ sql """drop table if exists ${aliasInOuter}"""
+ sql """
+ CREATE TABLE ${aliasInTable} (id INT, x INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ CREATE TABLE ${aliasInOuter} (id INT, k INT) DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO ${aliasInTable} VALUES (1, 5)"""
+ sql """INSERT INTO ${aliasInOuter} VALUES (1, 5)"""
+ sql """
+ CREATE MATERIALIZED VIEW ${aliasInMv}
+ BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT o.id FROM ${aliasInOuter} o
+ WHERE o.k IN (SELECT d.flag FROM (SELECT c.x AS flag FROM
${aliasInTable} c) d)
+ """
+ waitingMTMVTaskFinishedByMvName(aliasInMv)
+ order_qt_alias_in_subquery_baseline "SELECT id FROM ${aliasInMv}"
+ sql """ALTER TABLE ${aliasInTable} ADD COLUMN flag INT DEFAULT 0"""
+ order_qt_alias_in_subquery_state "select
Name,State,RefreshState,SyncWithBaseTables from
mv_infos('database'='${dbName}') where Name='${aliasInMv}'"
+ order_qt_alias_in_subquery_rows "SELECT id FROM ${aliasInMv}"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
