morningman opened a new pull request, #69772:
URL: https://github.com/apache/doris/pull/69772
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
**In short.** Once a base table's `changelog.ttl_seconds` expires the offset
of an IVM materialized view's internal stream, every refresh of that MV fails,
including refreshes that are allowed to fall back to COMPLETE, until someone
runs a manual COMPLETE refresh. This PR classifies that failure as
`CHANGELOG_BROKEN`, so a refresh that may fall back rebuilds the MV with
COMPLETE (the only attempt that resets the stream baselines) and a strict
`INCREMENTAL` refresh fails with that reason.
**Background.** An IVM MV reads each unique-key (merge-on-write) base table
through an internal table stream created with `MIN_DELTA`. Planning a
`MIN_DELTA` read refuses an offset that is older than what the table still
retains: `ChangelogUtils.effectiveStartTso` throws `Changelog offset has
expired according to changelog.ttl_seconds`. The incremental refresh turns a
failure into a fallback only when `IvmFailureClassifier` recognizes its message.
**The problem, and what it cost.**
- The expired-offset message was not one the classifier knew, so
`IvmIncrRefreshManager.doRefresh` rethrew it and the task failed even for
`REFRESH ... AUTO`, `INCREMENTAL FALLBACK`, or the scheduled refresh of a
`REFRESH INCREMENTAL FALLBACK` MV.
- A failed refresh does not move the offset, so while the TTL stays as it is
every later refresh fails the same way, and the MV keeps serving old rows.
- The delta is not the only read of that offset. When some MV partitions
need a rebuild (for example after a `TRUNCATE` of a base partition), the
incremental attempt first rebuilds them with a partition refresh, which reads
the tables the MV does not partition by through their streams at the offsets
(`StreamReadMode.SNAPSHOT`). With such an offset expired that rebuild failed
the same way, before the delta ran.
- `IvmFailureReason.CHANGELOG_BROKEN` had no producer in production code,
although `MTMVTask.handleIvmFallbackResult` already sends it straight to
COMPLETE; the comment and log of that branch described it as "previous run
incomplete".
**How this PR fixes it.**
- `IvmFailureClassifier` maps a failure carrying
`ChangelogUtils.CHANGELOG_OFFSET_EXPIRED` to `CHANGELOG_BROKEN` (matching the
constant the read throws with, not a copy of it).
- `MTMVTask.executeIvmAttempt` answers that failure of the rebuild before
the delta through `handleIvmFallbackResult`, the way the delta's failure is
answered; any other rebuild failure still fails the attempt.
- The `CHANGELOG_BROKEN` branch of `handleIvmFallbackResult` now explains
why it goes directly to COMPLETE (a partition refresh still reads the streams
at their offsets) and logs the detail with the offset and cutoff TSO.
**Results** (local shared-nothing cluster):
| Refresh after the offset expired | Before | After |
| --- | --- | --- |
| `REFRESH ... INCREMENTAL` (strict) | FAILED, `IvmFallbackReason` empty |
FAILED, `IvmFallbackReason = CHANGELOG_BROKEN`, error says
`reason=CHANGELOG_BROKEN` with both TSOs |
| `REFRESH ... AUTO` | FAILED, MV keeps the old rows | SUCCESS through
COMPLETE, `IvmFallbackReason = CHANGELOG_BROKEN` |
| the next `REFRESH ... AUTO` | fails again while the TTL is unchanged |
SUCCESS, incremental |
| `REFRESH ... AUTO` that first rebuilds a dirty MV partition while a joined
table's offset has expired | FAILED | SUCCESS through COMPLETE,
`IvmFallbackReason = CHANGELOG_BROKEN` |
**Classes, and how they call each other.**
- `MTMVTask.run` → `executeIvmAttempt` → (dirty partitions)
`executePartitionBasedRefresh` → on an expired offset
`handleIvmFallbackResult(CHANGELOG_BROKEN)`
- `executeIvmAttempt` → `executeSingleIvmAttempt` →
`IvmIncrRefreshManager.doRefresh` →
`IvmFailureClassifier.classifyExecutionFailure` →
`IvmIncrRefreshResult.fallback(CHANGELOG_BROKEN)` → `handleIvmFallbackResult`
- `handleIvmFallbackResult`: strict request → `JobException` with the
reason; otherwise `FALLBACK_TO_COMPLETE` → `executeCompleteAttempt`, which
resets the stream baselines.
Not changed: the planner's TTL check itself, the PARTITIONS attempt (it
reads a changed table at an old offset only after a partition refresh that
failed part-way: a plan covering every partition reads the joined tables with
`RESET`), and the persisted task fields.
### Release note
Fix an IVM materialized view refresh failing on every run after a base
table's `changelog.ttl_seconds` expired the stream offset. A refresh that
allows fallback now rebuilds the MV with COMPLETE; a strict INCREMENTAL refresh
fails with reason `CHANGELOG_BROKEN`.
### Check List (For Author)
- Test: Regression test / Unit Test
- Regression test: `mtmv_p0/ivm/test_ivm_changelog_offset_expired` (new,
nonConcurrent), covering the delta and the rebuild before it. Local
shared-nothing cluster: the delta case was first run on the base commit (strict
and AUTO both FAILED); the rebuild case was reproduced by hand on a build with
only the classifier change (AUTO FAILED on the rebuild). With the whole fix the
suite passes twice in a row against the generated `.out`. Not run on a cloud
cluster.
- Unit Test: `IvmIncrRefreshManagerTest` (new case), `MTMVTaskTest` (two
new cases, strict and fallback), `IvmFailureReasonTest`, `ChangelogUtilsTest`:
94 tests pass; each new case fails when its half of the fix is removed.
- Behavior changed: Yes. A refresh that allows fallback recovers with
COMPLETE instead of failing; a strict INCREMENTAL refresh reports
`CHANGELOG_BROKEN`.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
