yujun777 opened a new pull request, #68646:
URL: https://github.com/apache/doris/pull/68646
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68390
Problem Summary:
A column change on a base table (`DROP COLUMN` / `RENAME COLUMN`) puts every
materialized view that reads the table into `SCHEMA_CHANGE`, whether or not the
column is one the view's query names. That state is what a strict `REFRESH ...
INCREMENTAL` on an IVM MV reads -- the refresh is escalated to a whole-MV
`COMPLETE` for a column the MV never read -- and it is what stops the
transparent rewrite serving a plain MV until a refresh has run. Neither is work
the change owes.
The judgement such a change needs is the view's own query, per view: a
column that is dropped or renamed either is named by that query -- and then the
query no longer analyses, which is the state the view has to be left in -- or
it is not named, and the rows the view holds are the ones they should be.
`MTMVRelationManager` already re-analyses the query for such a change, but only
to pick a more specific reason for a record it writes either way. This PR makes
that answer the criterion, for both kinds of MV:
* Each operation says whether the query decides it
(`AlterOp#needQueryUsabilityCheck`). `DROP COLUMN` and `RENAME COLUMN` do. A
type change does not: the query keeps analysing while the meaning of the rows
computed under the old type changes, so re-analysis says nothing for a column
the query only filters or joins on. A rename of the table itself is not asked
about either, for the reason it never was: the view's query spells the old
name, so it is unanalysable by construction, and the query check could only
attribute the invalidation to the wrong reason. `REPLACE TABLE`, a dropped
table and a base view change were never asked about, and they still invalidate
every view that reads them.
* A change that has not reached the table is not asked about
(`AlterOp#hasReachedTheTable`). A schema change that is not a light one is
applied by a job, and the MV hook runs where that job may not have run yet: the
table still holds the column the change takes away, every query still analyses
against it, and an invalidation decided on that answer would be about the table
from before the change -- while the ABA that a dropped and re-added column
leaves behind is exactly what the invalidation exists for. What is asked is
whether the change has reached the table, which is the same fact the
re-analysis reads, so the two answers cannot disagree; a change that has not
been seen that way keeps invalidating, as it did before the queries were asked
at all.
Alongside it, two things the same code owed:
* `MTMV.processBaseViewChange` applied an invalidation in a second copy of
what `MTMV.alterStatus` does (state, version bump, snapshot drop). It goes
through `alterStatus` now, so an invalidation is one place to read whatever
carried it. The view path keeps applying without a record of its own, and that
is now written down where it happens: the change to the view is what is
journaled and a replay runs this hook along with it, so a record here would
apply the same invalidation twice on a follower -- including a version bump the
leader moved once.
* `MTMVState.SCHEMA_CHANGE` now says what it means, which is more than its
name: the state is "this MV needs a whole rebuild", and the comment names the
changes that set it -- for a plain MV as well as an IVM one, and for an IVM MV
alone -- and why the constant is not renamed (its name is what is written to
disk, and a constant Gson does not know reads back as null, dropping the state
of every MV loaded from such an image instead of failing).
An entire-table `TRUNCATE TABLE` is deliberately left as it is. The design
asked for it to be recorded as a whole-MV change; the per-partition requirement
already expresses that scope (every MV partition that reads the table is
named), and the whole-MV record would send every partition of the MV to a
rebuild and reset the streams for a change the per-partition requirement
carries -- where a stream does need that reconcile, the refresh falls back to a
`COMPLETE` on its own. Both truncate scopes are pinned by tests now.
Behaviour changed: Yes
* A base table `DROP COLUMN` / `RENAME COLUMN` no longer invalidates a
materialized view whose query does not name the column. The view keeps its rows
and stays a rewrite candidate, and a strict `REFRESH ... INCREMENTAL` on an IVM
MV is no longer escalated to a whole-MV `COMPLETE` because of it. A column the
query does name invalidates the view as before, and so does every other base
table change.
### Release note
Dropping or renaming a base table column no longer invalidates a
materialized view whose query does not use that column: the view keeps its
rows, keeps being used by the transparent rewrite, and a strict `REFRESH ...
INCREMENTAL` on an incremental materialized view no longer runs as a whole-view
`COMPLETE` refresh because of it. A column the view's query does use
invalidates the view exactly as before.
### Check List (For Author)
- Test: Unit Test / Regression test
- FE unit tests: `IvmBaselineRebuildTest` (42 -> 47),
`MTMVRelationManagerTest` (6 -> 7), `MTMVSchemaChangeTest`, `MTMVTest`,
`MTMVTaskTest`, `CreateMTMVCommandTest` -- 275 tests, all pass. Every new case
was checked to fail when the change it covers is reverted: the unreferenced
column leaves the MV in `SCHEMA_CHANGE` without the criterion, the flag is not
consulted without the hook change, the three effects of a view invalidation
stop moving together without the convergence, and an entire-table truncate
recorded as a whole-MV change fails the two truncate cases.
- Regression: `test_drop_unreferenced_column_mtmv` (new: an MV whose
partitions follow the base table's, and one on a table whose drops a job
rewrites), `test_base_drop_col_multi_level_mtmv` and
`test_base_rename_col_multi_level_mtmv` (one golden row each: the MV whose
query does not name the column is no longer invalidated, and the plan chooses
it where it previously was not a candidate),
`test_base_alter_col_type_multi_level_mtmv`, `test_base_recreate_col_mtmv`,
`test_create_mtmv_with_view`, `test_create_mtmv_with_view_alter`,
`test_create_view_mtmv`, `test_create_mtmv_with_view_pct`,
`test_ivm_drop_referenced_column_baseline_rebuild` (the unreferenced column no
longer escalates the refresh, the referenced one still does and its ABA case is
still rebuilt), `test_ivm_drop_column_fallback_reason`,
`test_ivm_row_binlog_schema_validation`, `test_ivm_baseline_marker_scope`,
`test_ivm_partition_epoch_rebuild`, `test_ivm_partition_baseline_rebuild`,
`test_ivm_unconsumed_dimension_i
s_not_current` -- all pass; the two goldens that changed were regenerated and
reviewed line by line.
- Behavior changed: Yes (see above)
- Does this need documentation: Yes, in a follow-up (no doc PR yet): the
user-facing page's list of situations that leave a materialized view behind is
tracked in the design document-change list, which records the exact locations
and wording for this and for the notes the previous PR in the series owes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
