morningman commented on PR #69778:
URL: https://github.com/apache/doris/pull/69778#issuecomment-6474415651
### Steps to reproduce
One unique-key table, one non-partitioned IVM MV, one BE. The FE needs
`enable_feature_changelog = true` and `enable_table_stream = true` in `fe.conf`
(the regression pipelines set both).
```sql
CREATE DATABASE IF NOT EXISTS ivm_repro;
USE ivm_repro;
-- Keep the queries on t answered by t itself; step 6 turns this back on.
SET enable_materialized_view_rewrite = false;
DROP MATERIALIZED VIEW IF EXISTS mv;
DROP TABLE IF EXISTS t;
CREATE TABLE t (
k BIGINT NOT NULL,
dt DATE NOT NULL,
v BIGINT
)
UNIQUE KEY(k, dt)
PARTITION BY RANGE(dt) (
PARTITION p202601 VALUES [('2026-01-01'), ('2026-02-01')),
PARTITION p202602 VALUES [('2026-02-01'), ('2026-03-01'))
)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"enable_unique_key_merge_on_write" = "true",
"changelog.enable" = "true",
"changelog.need_historical_value" = "true",
"changelog.ttl_seconds" = "86400"
);
INSERT INTO t VALUES (1, '2026-01-10', 10); -- p202602 stays empty for now
-- 1. Build the MV. Its first refresh is a COMPLETE one: p202601 gets a
stream offset, the empty p202602 gets none.
CREATE MATERIALIZED VIEW mv
BUILD IMMEDIATE REFRESH INCREMENTAL ON MANUAL
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES ("replication_num" = "1")
AS SELECT k, dt, v FROM t;
-- repeat until Status is SUCCESS
SELECT Status, RefreshMode FROM tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime
DESC LIMIT 1;
-- 2. Rows in the partition that was empty and in a partition added after
the MV: neither partition has an offset.
ALTER TABLE t ADD PARTITION p202603 VALUES [('2026-03-01'), ('2026-04-01'));
INSERT INTO t VALUES (2, '2026-02-10', 20), (3, '2026-03-10', 30);
-- 3. Let those rows grow older than the TTL before the MV reads them. This
stands in for a refresh that comes
-- more than a day after the write with the default TTL of one day.
ALTER TABLE t SET ("changelog.ttl_seconds" = "1");
SELECT sleep(2);
-- 4. A strict incremental refresh; repeat the query until the task has
finished.
REFRESH MATERIALIZED VIEW mv INCREMENTAL;
SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg FROM
tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime
DESC LIMIT 1;
-- 5. Compare the MV with the table.
SELECT * FROM mv ORDER BY k;
SELECT * FROM t ORDER BY k;
-- 6. With transparent rewrite on, the same query on t is answered from mv.
SET enable_materialized_view_rewrite = true;
SELECT k, dt, v FROM t ORDER BY k;
EXPLAIN SHAPE PLAN SELECT k, dt, v FROM t ORDER BY k;
```
What it printed on a local shared-nothing cluster:
| Step | Without this PR (master `86d62ae00c9`, with #69785 on top, which
does not touch this path) | With this PR (`7549bc2bb21`) |
| --- | --- | --- |
| 1. build task | `SUCCESS`, `COMPLETE` | `SUCCESS`, `COMPLETE` |
| 4. refresh task | `SUCCESS`; `RefreshMode` and `IvmFallbackReason` are
`\N`, `ErrorMsg` is empty | the same |
| 5. `SELECT * FROM mv` | one row: `1, 2026-01-10, 10` | three rows: `1,
2026-01-10, 10` / `2, 2026-02-10, 20` / `3, 2026-03-10, 30` |
| 5. `SELECT * FROM t` | the three rows | the three rows |
| 6. the same query with rewrite on | one row; the plan is a
`PhysicalOlapScan[mv]` | three rows; the plan is a `PhysicalOlapScan[mv]` |
So without the fix the refresh reports success, the MV is short of both rows
for good (their partitions now have offsets past them), and the query on `t`
returns the MV's answer. Nothing in the task, its error or its fallback reason
says that anything went wrong.
Variants, all in
[`test_ivm_partition_without_offset`](https://github.com/apache/doris/blob/7549bc2bb2184cee197e66ee0c905c3d50f272e9/regression-test/suites/mtmv_p0/ivm/test_ivm_partition_without_offset.groovy):
a duplicate-key table with an append-only detail MV loses the same two rows;
an MV partitioned by `dt` rebuilds the MV partition of `p202603` but loses the
row of `p202602`, whose MV partition existed already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
