laskoviymishka opened a new pull request, #69135:
URL: https://github.com/apache/doris/pull/69135
### What problem does this PR solve?
Issue Number: #68916
Related PR: #69031
Problem Summary: The Iceberg REST catalog spec now lets a catalog return
`labels` with each table load (apache/iceberg#15750): catalog-owned,
per-request metadata such as ownership, classification or cost attribution, at
the table level (`object-labels`) and per field (keyed by `field-id`). Labels
are not part of the table metadata, have no snapshots and are never written to
`metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via
`SupportsLabels#labels()`. Doris had no way to show them.
Add a `t$labels` system table for iceberg tables in a REST catalog:
```sql
SELECT * FROM rest_ctl.db.tbl$labels;
scope | field_id | field_name | key | value
object | NULL | NULL | owner | team-a
field | 2 | email | classification | pii
```
One row per label. `field_name` is resolved against the current table schema
and is NULL for a field id that is no longer in it (a dropped column). The
schema mirrors iceberg's proposed `LabelsTable`.
Implementation:
- Keep labels when Doris rebuilds a loaded table. iceberg's REST client
attaches them to the loaded `BaseTable` as a transient field; they are not
table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass
them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached
reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and
`IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used
constructors that reset labels to `Labels.EMPTY`, so every read saw no labels;
they now pass `baseTable.labels()`.
- `labels` is not a `MetadataTableType`, so it is handled next to the SDK
metadata tables rather than through `MetadataTableUtils`.
`IcebergConnectorMetadata` lists and resolves it only when the catalog flavor
is `rest` (no other catalog returns labels), and serves its fixed schema and
column handles without loading the base table.
- `IcebergScanPlanProvider` plans it as one split. A new split-package
helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg
`StaticDataTask` (package-private in the SDK) holding the label rows, projected
to the requested columns in BE's required-field order; its metadata `DataFile`
is synthesized, so no storage request is made. The task is serialized like
every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it
with no BE change.
- The ordered system-table projection is extracted into
`projectSystemTableSchema` and shared with the existing path, and the
base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`.
- Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin
of the handle is ignored, and `$labels` is exempt from the
resolved-empty-snapshot fence, so a table with no commit still shows its labels.
Labels reflect the table load the statement used, so they follow the iceberg
table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new
labels. Like the other system tables, a REST catalog that requires server-side
scan planning rejects `$labels`.
### Release note
Support querying REST catalog labels of an iceberg table through the
`tbl$labels` system table.
### Check List (For Author)
- Test: Unit Test
- Added IcebergLabelsDataTaskTest, the $labels cases in
IcebergConnectorMetadataSysTableTest,
IcebergTableCacheTest#snapshotReadTableKeepsCatalogLabels and
IcebergStatementScopeTest#sharedTableKeepsCatalogLabels.
- `build.sh --fe` (Maven build cache disabled) passes on top of current
master. Ran every iceberg and hudi FE unit test with run-fe-ut.sh: 1990 tests,
0 failures, 5 skipped.
- Regression test pending a REST fixture that returns labels
(apache/iceberg#18047).
- Behavior changed: No (adds a new system table for REST catalogs only)
- Does this need documentation: Yes (doc PR to follow)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
