morningman opened a new pull request, #69856:
URL: https://github.com/apache/doris/pull/69856
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
**In short.** A refresh of an IVM materialized view can keep the MV up
incrementally or quietly recompute it, and nothing tells the two apart:
`tasks("type"="mv")` does not say which kinds of work a task committed, and
`/metrics` has no IVM series at all. This PR makes each refresh task of an IVM
MV report how it refreshed the MV (`IvmRefreshMethod` in `tasks()`) and why it
failed or fell back, keeps a refresh summary on the MV for the IVM status table
that follows, and counts the tasks on the Master in five new counters.
**Background.** An IVM MV is refreshed by a task that runs one or more
attempts: the incremental attempt (the delta of its base tables' streams, after
rebuilding the MV partitions an invalidation marked), a partition rebuild, and
a whole-MV rebuild. A request that may fall back (AUTO, INCREMENTAL FALLBACK,
PARTITIONS FALLBACK) moves on to the next attempt when one gives up, for
example to COMPLETE after the delta meets a MIN/MAX boundary. `tasks()` already
reports `RefreshMode` (which partitions the task covered), `IvmFallbackReason`
(why the incremental attempt gave up, or why a strict request failed) and
`IvmRebuiltPartitions`.
**The problem, and what it cost.**
- An operator cannot tell a refresh that kept the MV up incrementally from
one that recomputed it. `RefreshMode` says `COMPLETE` for any whole-MV scope,
and `IvmFallbackReason` is set even when the attempt that replaced the delta
wrote nothing, or when a strict request failed.
- No alert can watch IVM refreshes. The asynchronous MV counters
(`async_materialized_view_task_success_num`, `_failed_num`, `_skip_num`) cover
every MV and say nothing about the method, the fallbacks, or why tasks fail. An
MV that falls back to COMPLETE on every run, the expensive failure mode of IVM,
goes unnoticed until the load of the cluster shows it.
- A task that the full MV task queue turns away never runs, and nothing
counts it.
- The task history keeps only `max_persistence_task_count` tasks, so how
many times in a row an MV has failed, or when it last fell back, cannot be read
back from it.
**How this PR fixes it.**
- `MTMVTask` notes each batch that commits data (the delta, a partition
rebuild, a whole-MV rebuild) and decides `IvmRefreshMethod` once, as the task
publishes its result: `NONE`, `INCREMENTAL`, `PARTITION_REBUILD`, `COMPLETE`,
or `MIXED` when more than one kind committed, typically the rebuild of
invalidated partitions before the delta. `tasks("type"="mv")` gets the
`IvmRefreshMethod` column, NULL while a task runs and for a task of an MV that
is not an IVM MV.
- A task falls back when the incremental attempt gives up and an attempt
replacing it commits data. Not fallbacks: building the first baseline of an MV
no refresh has succeeded on, the rebuild of invalidated partitions before the
delta, an explicit COMPLETE or PARTITIONS request, and a fallback that commits
nothing.
- A failed task reports why: the `IvmFailureReason` a request that may not
fall back failed with, or else the stage it failed in, `QUERY_UNUSABLE`,
`PARTITION_SYNC_FAILED`, `EXECUTION_FAILED` or `OTHER`, told apart by where the
failure was thrown rather than by its message.
- The method, the failure reason and whether the task fell back are
journaled with the task, as new Gson fields that an older FE ignores. The MV
keeps an `IvmRefreshSummary`, updated from each task result in
`MTMV.addTaskResult`, which the Master and the replay of the same journal
record run alike, from what the record carries: consecutive failures, the
latest success and failure (with the reason and the error message, cut to 512
characters), the latest method and run time, and the fallback count with the
latest reason. A canceled task leaves it as it is. Nothing reads it in this PR;
`information_schema.ivm_mv_status` and the per-MV series of a follow-up PR do.
- Five counters, counted on the Master as each task of an IVM MV ends. Every
label combination is registered up front, so each of the 61 series exists from
0:
| Series | Type | Labels | Value |
| --- | --- | --- | --- |
| `doris_fe_ivm_refresh_tasks_total` | counter |
`result`=success/no_change/failed/canceled,
`method`=none/incremental/partition_rebuild/complete/mixed | refresh tasks that
ended; `no_change` is a SUCCESS that committed nothing |
| `doris_fe_ivm_refresh_duration_seconds_total` | counter | `method` | the
run time, StartTime to FinishTime, of the tasks that succeeded or failed; a
canceled task has no finish time |
| `doris_fe_ivm_fallbacks_total` | counter | `reason`=the `IvmFailureReason`
values, `incomplete_refresh_snapshot` | tasks that fell back |
| `doris_fe_ivm_refresh_failures_total` | counter | `reason`=the
`IvmFailureReason` values, the four stages | tasks that failed |
| `doris_fe_ivm_schedule_rejected_total` | counter | | tasks the full MV
task queue turned away |
Like the other FE event counters they live in the registry of every FE and
only the Master counts in them, so a query sums them over the FEs; a restart or
a new Master starts them from 0.
- `AbstractTask.onRejected`, a new hook that `TaskProcessor.addTask` calls
when the executor queue rejects a task. Both the instant dispatch
(`JobScheduler`) and the timer dispatch (`DispatchTaskHandler`) add tasks
through it, and `MTMVTask` counts the rejection for an IVM MV. A
timer-dispatched task that is turned away is canceled as before, so it also
counts as `canceled`.
**Results.** Before this PR, `tasks()` has no method column and `/metrics`
has none of these series. After it, on a local cluster, one IVM MV refreshed
through each case (regression `test_ivm_refresh_method_metrics`):
| Refresh | `tasks()`: Status, IvmRefreshMethod, IvmFallbackReason |
Counters that moved, besides the run time |
| --- | --- | --- |
| first AUTO refresh of a new MV | SUCCESS, COMPLETE, NULL |
`tasks{success,complete}` |
| INCREMENTAL after an insert | SUCCESS, INCREMENTAL, NULL |
`tasks{success,incremental}` |
| INCREMENTAL with nothing new | SUCCESS, NONE, NULL |
`tasks{no_change,none}` |
| INCREMENTAL after deleting the row that holds a group's minimum | FAILED,
NONE, MIN_MAX_BOUNDARY_HIT | `tasks{failed,none}`,
`failures{min_max_boundary_hit}` |
| INCREMENTAL FALLBACK in the same state | SUCCESS, COMPLETE,
MIN_MAX_BOUNDARY_HIT | `tasks{success,complete}`,
`fallbacks{min_max_boundary_hit}` |
| COMPLETE | SUCCESS, COMPLETE, NULL | `tasks{success,complete}` |
| INCREMENTAL after truncating a base partition and inserting into another |
SUCCESS, MIXED, NULL | `tasks{success,mixed}` |
The first refresh of a new MV finds every partition still to be filled and
goes straight to COMPLETE; it counts no fallback. The MV holds what its base
table does after each case.
**Classes, and how they call each other.**
- `IvmRefreshMethod`, `IvmRefreshFailureStage`
(`org.apache.doris.mtmv.ivm`): the methods, and the stages a failure is
reported by.
- `MTMVTask`: notes committed batches (`recordCommitted`), decides whether
giving up the incremental attempt is a fallback
(`givingUpTheIncrementalAttemptIsAFallback`), moves the failure stage along,
settles the method and the failure reason in `after()`, and counts the task
(`recordIvmRefreshMetrics`) and a rejection (`onRejected`).
- `IvmRefreshSummary` (`org.apache.doris.mtmv.ivm`): persisted on `MTMV` and
updated by `update(MTMVTask)` from `MTMV.addTaskResult`;
`MTMV.hasSucceededIvmRefresh` tells the first baseline apart.
- `MetricRepo`: registers the five counters. `DoubleCounterMetric` (new)
keeps a total in seconds without rounding each addition.
- `AbstractTask.onRejected` (new, empty by default), called by
`TaskProcessor.addTask`.
```
MTMVTask.run() worker thread
├─ executeIvmAttempt
│ ├─ rebuild invalidated partitions ──►
recordCommitted(PARTITION_REBUILD)
│ └─ delta (IvmIncrRefreshManager) ───► recordCommitted(INCREMENTAL)
│ gave up ─► fallingBack = givingUpTheIncrementalAttemptIsAFallback()
├─ PARTITIONS / COMPLETE attempts ───────► recordCommitted(...)
ivmFallback once fallingBack
└─ failureStage: QUERY_UNUSABLE ─► PARTITION_SYNC_FAILED ─►
EXECUTION_FAILED, or an IvmFailureReason
onSuccess / onFail / cancel ─► after()
├─ ivmRefreshMethod, ivmFailureReason settled
├─ recordIvmRefreshMetrics() ─► MetricRepo.COUNTER_IVM_* Master
only
└─ Env.addMTMVTaskResult ─► MTMV.addTaskResult ─►
IvmRefreshSummary.update(task)
(journaled as ADD_TASK; the replay calls
addTaskResult the same way)
TaskProcessor.addTask ── queue full ─► task.onRejected() ─► MTMVTask:
COUNTER_IVM_SCHEDULE_REJECTED
```
Nothing else changes: no thrift or proto change, no new config, and the
existing columns and series stay as they are.
### Release note
`tasks("type"="mv")` has a new column `IvmRefreshMethod`: how a refresh task
of an IVM materialized view refreshed it (NONE, INCREMENTAL, PARTITION_REBUILD,
COMPLETE or MIXED). New FE metrics
`doris_fe_ivm_refresh_tasks_total{result,method}`,
`doris_fe_ivm_refresh_duration_seconds_total{method}`,
`doris_fe_ivm_fallbacks_total{reason}`,
`doris_fe_ivm_refresh_failures_total{reason}` and
`doris_fe_ivm_schedule_rejected_total`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit test: `MTMVTaskTest` checks the method each kind of committed batch
gives, that the rebuild of invalidated partitions before the delta is `MIXED`
and no fallback, that only what commits after the incremental attempt gave up
makes a fallback, that building the first baseline is not one, the stage a
failure reports (query, partition sync, execution) and the IVM reason of a
strict request, the `IvmRefreshMethod` column, and that the report is journaled
while what it is decided from is not. `IvmRefreshSummaryTest` checks
consecutive failures across success, no change, failure and cancel, the latest
success and failure, the error message cut, and the fallback count. `MTMVTest`
checks that the Master and the replay of the same journal records end with the
same summary with a task history of one, and that an MV that is not an IVM MV
keeps none. `IvmRefreshMetricsTest` checks that every series is there from 0
and how each result is counted; `TaskProcessorTest` that only the task
the full queue rejects is told so. The mtmv, IVM, metric and job executor
tests with `CreateMTMVCommandTest`, `ShowCreateMTMVTest` and
`DropMaterializedViewTest` pass: 60 classes, 884 cases.
Regression test: `mtmv_p0/ivm/test_ivm_refresh_method_metrics`
(nonConcurrent: it asserts which counters of the Master moved), twice on a
local shared-nothing cluster; not run in cloud mode.
Manual: `promtool check metrics` on the `/metrics` of the Master reports
no parsing error and no lint finding on the new series.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. The doris-website change comes with the documentation of the
changelog and IVM monitoring.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
