morningman opened a new pull request, #67010:
URL: https://github.com/apache/doris/pull/67010
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Background jobs cannot be pinned to a specific compute group today. A
routine load
job silently snapshots whatever compute group the creating session happened
to be
on, and an async materialized view has no compute group property at all:
automatic
refreshes run wherever `admin` resolves to, while a manual `REFRESH` borrows
the
triggering session's group. So the same MV can refresh in two different
places
depending on who triggered it, and there is no way to isolate one job's
resources
from another's.
This PR adds a `compute_group` property to `CREATE`/`ALTER ROUTINE LOAD` and
to
async materialized views, so a job can be pinned explicitly.
This is a transitional binding ahead of the full
`(owner, compute_group, workload_group)` model. The property name and its
value
space are deliberately identical to that design, and the key is written only
when
the user actually declared it, so metadata written here is read back as an
explicit
pin later on with no conversion:
- the declaration lives in the already existing property maps
(`RoutineLoadJob.jobProperties`, `MTMV.mvProperties`), so no new persisted
field,
no journal type and no `FeMetaVersion` bump is needed;
- an absent key keeps the existing implicit resolution untouched, which is
also how
a pinned group is told apart from an inherited one;
- `DEFAULT` is rejected because it is reserved to mean "follow the owner's
default
group", and a group literally named `DEFAULT` would otherwise be silently
reinterpreted after an upgrade;
- non-cloud is out of scope for now and rejected at `CREATE`/`ALTER`, so
non-cloud
metadata never carries the key.
**Routine load** resolves the effective cluster in `getCloudCluster()`,
which every
consumer already goes through: backend selection, the thrift task sent to
BE, the
workload group namespace, the plan context used by `OlapTableSink`, and the
`ComputeGroup` column of `SHOW ROUTINE LOAD`. `ALTER` only updates the
property map
and is covered by the existing edit log, so replay needs no change.
**Async MVs** resolve it in `MTMVTask.setComputeGroup()`, the single point
where the
group is applied, so every trigger path is covered including the scheduler
one. A
declared group also wins over the triggering session for a manual `REFRESH`,
so the
same MV no longer refreshes in two different places depending on who
triggered it.
MVs that declare nothing keep borrowing the session's group, so existing MVs
are
unaffected.
Two correctness fixes fall out of this:
- `CREATE ROUTINE LOAD` validated the workload group against the session's
compute
group. Since a workload group lives in a compute group's namespace, that
check has
to run against the declared group, otherwise a valid combination is
rejected and an
invalid one only fails on the first task.
- `RoutineLoadJob.plan()` set the plan context from the raw snapshot field
rather
than the getter, which would have sent the write path to the creating
session's
group while the tasks ran in the declared one.
Usage:
```sql
CREATE ROUTINE LOAD db.job ON tbl
COLUMNS TERMINATED BY ","
PROPERTIES ("compute_group" = "cg_etl")
FROM KAFKA (...);
ALTER ROUTINE LOAD FOR db.job PROPERTIES ("compute_group" = "cg_etl_2");
CREATE MATERIALIZED VIEW mv
BUILD DEFERRED REFRESH AUTO ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 2
PROPERTIES ("replication_num" = "1", "compute_group" = "cg_batch")
AS SELECT k1, k2 FROM tbl;
ALTER MATERIALIZED VIEW mv SET ("compute_group" = "cg_batch_2");
```
### Release note
Support declaring `compute_group` on routine load jobs and async
materialized views
in cloud mode, so a background job can be pinned to a specific compute group
instead
of inheriting one implicitly.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. A materialized view that declares `compute_group` now
refreshes in
that group even when a manual `REFRESH` is triggered from a session on
a
different compute group. MVs that do not declare the property are
unaffected
and keep borrowing the session's group.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new `compute_group` property on routine load and async MV
needs a
doc entry, including that it is cloud-mode only for now and that
`DEFAULT` is
reserved.
### Check List (For Reviewer who merge this PR)
- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
