yujun777 opened a new pull request, #70785: URL: https://github.com/apache/doris/pull/70785
### Background Incremental view maintenance stores a signature of the normalized query plan to check that a refresh can find and update the rows written by earlier refreshes. Creating a view normalizes a query with a result sink. Refreshing the view normalizes an INSERT with a table sink. ### What problem does this PR solve? Issue Number: None Related PR: #62606 (introduced IVM layout signatures), #69890 (publishes the signature after a complete refresh) PR Type: Bugfix Problem Summary: A deferred aggregate IVM view can fail signature validation before its first complete refresh. One example groups a UNION ALL over a VARCHAR column and a shorter string literal. The union yields STRING, and the table sink truncates it when writing to the bounded VARCHAR column of the view. This PR makes a write-only truncation project transparent to the signature. ### Details - **Frequency:** Always when the table sink adds a write-only string truncation project to a query that otherwise has a matching IVM layout. - **Root cause:** IvmPlanSignatureGenerator treated the sink's substring projection as a change to the maintenance plan, although it only adapted a value after the row identifier and aggregate state had been computed. - **Why existing tests missed it:** IvmPlanSignatureGeneratorTest covered passthrough outputs and sink bookkeeping columns. The signature fallback regression tested an injected signature mismatch after a complete refresh. Neither covered write truncation before the first complete refresh. - **Fix:** Walk the contiguous projects under the OLAP table sink and check that each projected value is a passthrough or a truncation matching its target column. Keep the sink's existing hidden defaults transparent. Preserve the signature for truncation in the query and for changes to persisted maintenance columns. - **Other paths and branches:** The unique-key table sink uses the same canonicalization for full and incremental refreshes. Append-only views do not generate layout signatures. The signature generator is absent in the local upstream 3.1, 4.0, 4.1, and 4.2 branch refs. - **Behavior change:** A write-only string truncation no longer changes the IVM layout signature for a newly created view. ### Testing The original aggregate view definition fails EXPLAIN ANALYZED PLAN REFRESH MATERIALIZED VIEW INCREMENTAL WITH ALL STREAMS on the baseline before any refresh task. With the fix, the same EXPLAIN succeeds before any refresh task, and its write plan still contains the truncation. A subsequent complete refresh and strict incremental refresh both succeed. The four result groups match the original base-table query. The FE build and all 26 tests in IvmPlanSignatureGeneratorTest passed locally. ### Release note Deferred incremental materialized views no longer report a layout mismatch solely because the table sink truncates a visible string column at write time. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
