morningman commented on PR #69785:
URL: https://github.com/apache/doris/pull/69785#issuecomment-6484177734
### Steps to reproduce
One duplicate-key table, one IVM MV over it (a plain projection, so an
append-only detail MV), one BE. The FE needs `enable_feature_changelog = true`
and `enable_table_stream = true` in `fe.conf` (the regression pipelines set
both).
#69772 already rejects an expired offset for a unique-key base table, whose
internal stream is read `MIN_DELTA`. A duplicate-key table is read
`APPEND_ONLY`, and its expired offset is clipped instead; that is the case
below.
```sql
CREATE DATABASE IF NOT EXISTS ivm_repro;
USE ivm_repro;
-- Keep the queries on t answered by t itself; step 7 turns this back on.
SET enable_materialized_view_rewrite = false;
DROP MATERIALIZED VIEW IF EXISTS mv;
DROP TABLE IF EXISTS t;
CREATE TABLE t (
k BIGINT NOT NULL,
v BIGINT
)
DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"changelog.enable" = "true",
"changelog.ttl_seconds" = "86400"
);
INSERT INTO t VALUES (1, 10);
-- 1. Build the MV. Its first refresh is a COMPLETE one and records the
offset of the MV's stream on t.
CREATE MATERIALIZED VIEW mv
BUILD IMMEDIATE REFRESH INCREMENTAL ON MANUAL
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES ("replication_num" = "1")
AS SELECT k, v FROM t;
-- repeat until Status is SUCCESS
SELECT Status, RefreshMode FROM tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime
DESC LIMIT 1;
-- 2. A row written after the offset.
INSERT INTO t VALUES (2, 20);
-- 3. Let the offset, and the row with it, grow older than the TTL before
the MV reads them. This stands in
-- for a refresh that comes more than a day after a write, with the
default TTL of one day.
ALTER TABLE t SET ("changelog.ttl_seconds" = "1");
SELECT sleep(2);
-- 4. A strict incremental refresh; repeat the task query until the task has
finished.
REFRESH MATERIALIZED VIEW mv INCREMENTAL;
SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg FROM
tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime
DESC LIMIT 1;
SELECT * FROM mv ORDER BY k;
-- 5. A refresh that may fall back to COMPLETE; repeat the task query until
the task has finished.
REFRESH MATERIALIZED VIEW mv AUTO;
SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg FROM
tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime
DESC LIMIT 1;
-- 6. Compare the MV with the table.
SELECT * FROM mv ORDER BY k;
SELECT * FROM t ORDER BY k;
-- 7. With transparent rewrite on, the same query on t is answered from mv.
SET enable_materialized_view_rewrite = true;
SELECT k, v FROM t ORDER BY k;
EXPLAIN SHAPE PLAN SELECT k, v FROM t ORDER BY k;
```
What it printed on a local shared-nothing cluster:
| Step | Without this PR (master `86d62ae00c9`, the parent of this PR) |
With this PR (`0d4e500f26b`) |
| --- | --- | --- |
| 1. build task | `SUCCESS`, `COMPLETE` | `SUCCESS`, `COMPLETE` |
| 4. strict refresh task | `SUCCESS`; `RefreshMode` and `IvmFallbackReason`
are `\N`, `ErrorMsg` is empty | `FAILED`; `RefreshMode` is `\N`,
`IvmFallbackReason` is `CHANGELOG_BROKEN`, `ErrorMsg` is `IVM incremental
refresh failed for mv=mv, reason=CHANGELOG_BROKEN,
detail=org.apache.doris.nereids.exceptions.AnalysisException: Changelog offset
has expired according to changelog.ttl_seconds: start_tso=..., cutoff_tso=...` |
| 4. `SELECT * FROM mv` | one row: `1, 10` | one row: `1, 10` |
| 5. AUTO refresh task | `SUCCESS`; `RefreshMode` and `IvmFallbackReason`
are `\N`, `ErrorMsg` is empty | `SUCCESS`, `COMPLETE`; `IvmFallbackReason` is
`CHANGELOG_BROKEN`, `ErrorMsg` is empty |
| 6. `SELECT * FROM mv` | one row: `1, 10` | two rows: `1, 10` / `2, 20` |
| 6. `SELECT * FROM t` | the two rows | the two rows |
| 7. the same query with rewrite on | one row; the plan is a
`PhysicalOlapScan[mv]` | two rows; the plan is a `PhysicalOlapScan[mv]` |
So without the fix the strict refresh reports success without the row
written in step 2, and the AUTO refresh after it does not fall back either. No
later incremental refresh brings the row back (the stream offset is now past
it), and the query on `t` returns the MV's answer. Nothing in either task, its
error or its fallback reason says that anything went wrong. With the fix the
strict refresh fails with `CHANGELOG_BROKEN` and leaves the MV as it was, and
the AUTO refresh rebuilds it with COMPLETE.
Variants, all in
[`test_ivm_changelog_offset_expired_dup`](https://github.com/apache/doris/blob/0d4e500f26bfba4918e20386eedb39784065699d/regression-test/suites/mtmv_p0/ivm/test_ivm_changelog_offset_expired_dup.groovy):
an aggregate MV over the same table behaves as the detail MV here, without the
fix and with it; a user stream on the table still reads from the first TSO the
changelog retains; and once the TTL is back to a day, the refresh after the
rebuild is incremental again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
