yujun777 opened a new issue, #68767: URL: https://github.com/apache/doris/issues/68767
### Version master (and any branch carrying the MTMV query judgement, #68646). Found while working on the IVM series tracked by #65418. ### What's Wrong? A light base table schema change becomes visible to the query planner **before** the materialized views that read the table are invalidated, and nothing in the rewrite's eligibility check notices that the two are out of step. For a light schema change the new schema is installed inside the base table's write lock, and the MTMV hook that puts the views reading the table into `SCHEMA_CHANGE` runs only after that lock has been released: - `SchemaChangeHandler.process` holds `olapTable.writeLockOrDdlException()` for its whole body, calls `modifyTableLightSchemaChange(...)` (which installs the new column) and releases the lock at the end of the method -- `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:2677-2701`. - Back in the caller, the hook runs after that lock is gone -- `fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java:376-400` (`Env.getCurrentEnv().getMtmvService().alterTable(...)`). In that window a query can be planned against the new schema and transparently rewritten to a materialized view that is still `NORMAL`, whose rows were computed under the pre-change binding of the name it moved. A light schema change does not bump the base table's visible version, and the view's snapshot is unchanged, so `SyncWithBaseTables` stays true and the rewrite's checks pass. Nothing in the hook itself is wrong here; the deficit is that between "the schema is visible" and "the views that read it have been judged" there is an interval in which the rewrite still trusts the views. Two things widen or move the window: - The query judgement added in #68646 re-analyses each dependent view's query inside the hook before deciding, so the interval grows from "one state change per dependent view" to "one query analysis per dependent view". On master the same window exists, just narrower. - Observers have the same interval through the journal, and it is not bounded by an analysis: the light schema change is journaled first and the view invalidation as its own record after it, so a replayed (already published) schema is visible to queries on an observer before the invalidation record has been applied there. ### What You Expected? The transparent rewrite must not serve a materialized view in the interval between a base table schema change being published and the views that read it being invalidated -- either the invalidation is established before the new schema becomes visible, or the rewrite refuses those views until they have been judged against the change. ### How to Reproduce? The window is an interval between two statements of one DDL, so it is not a deterministic reproduction; it needs a planner running concurrently with a light DDL. The shape it bites in is a light change that makes a view's query bind a name to another column, e.g.: ```sql -- t1(k1 int, v1 int) with light schema change enabled -- mv1: SELECT k1, SUM(v1) AS total FROM t1 GROUP BY k1 WHERE flag IN (SELECT flag FROM t2) -- (a name the stored query reaches unqualified, so a column added to t1 answers for it from then on) ALTER TABLE t1 ADD COLUMN flag INT DEFAULT 0; -- light: published under the write lock, invalidated after ``` A query planned in the window binds `flag` to the new column and can be rewritten to `mv1`, whose rows were computed while `flag` was answered for by another column. What is verified today is the code path above (the publish and the invalidation are in two different lock scopes); the wrong result itself needs a concurrent planner inside the window and has not been reproduced deterministically. ### Anything Else? Options considered, with their costs: - **(A) A judgement barrier the rewrite respects** -- a transient marker on the views that read the table, set before the new schema is published and cleared once the views have been judged. It has to be a count rather than a flag (two concurrent ALTERs on the same table would clear each other's window) and it must be cleared in a `finally` (a leaked mark silently keeps a view out of the rewrite forever). It closes only the leader's window; the observer interval above needs the marker to be replayable or a different criterion. - **(B) Decide under the write lock, journal after** -- move the judgement into the lock scope that publishes the schema, and defer only the edit log waits (holding a table write lock across a journal wait is against the locking rules, and the query analysis would then hold the table write lock for every dependent view). - **(C) Compare a schema identity** -- have a view record the base table's schema identity when it refreshes, and have the rewrite require it to still match. This is the only one of the three that also closes the observer interval, and it is the largest: a new persisted field written by every refresh, plus replay and image compatibility. Related: #68646 (the judgement that widened the window), #65418 (the IVM tracker). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
