Gabriel39 opened a new pull request, #70776:
URL: https://github.com/apache/doris/pull/70776
### What problem does this PR solve?
Issue Number: None
Related PR: #70705 (merged master implementation; manually adapted to
branch-4.2), #70366 (only its shared codepoint-aware SQL replacement helper)
Background:
External catalog properties reach SHOW output, SQL audit rendering and
lineage metadata. Branch-4.2 uses a static `PrintableMap` registry rather than
master's connector provider registration. Its registry omits several typed
credential aliases and only examines property names, allowing retained session
tokens, JDBC connection strings and credentials embedded in otherwise ordinary
URLs to appear in output.
Problem Summary:
Bring the structured property masking behavior of #70705 to branch-4.2 while
preserving the branch's Java 8 API compatibility and its existing catalog
implementations. For example, formatting
`mc.session_token=fixture-session-material` with masking enabled previously
printed the value; it now prints `mc.session_token = *XXX`. Ordinary REST
endpoints remain readable, while embedded userinfo and secret parameters are
removed or masked. Runtime connection values remain intact.
Details:
- **Frequency:** Always for an affected retained property when a covered
output path renders it.
- **Root cause:** The legacy exact-name registry misses credential aliases
and has no value-aware address policy. ALTER CATALOG also lacks both the
audit-encryption interface and the property-range parser visitor on this
branch. Its shared SQL replacement helper additionally treats parser codepoint
positions as UTF-16 indices, which can truncate a masked range when SQL
contains supplementary characters.
- **Why existing tests missed it:** `PrintableMapTest` checks the older
exact-name list and basic formatting. The existing external-catalog audit
regression covers Iceberg OAuth fields but not MaxCompute/Ozone, JDBC URLs,
custom/Hadoop credentials or URL userinfo. Existing S3 streaming audit tests
check the selected batch but lack credential-bearing and literal-path boundary
cases.
- **Fix:** Keep a permanent credential union, add the same supplemental name
and structured-value rules as master, and route the branch's existing output
paths through it. Add both missing ALTER CATALOG audit hooks. Preserve selected
S3 object identities while masking each internal singleton before building its
display glob.
- **Other paths and branches:** Master is fixed by #70705; this PR targets
branch-4.2 only. The previous branch comparison also identified gaps on
branch-4.1; it is not changed here. The omissions originate in the legacy
registry; a single introducing PR has not been established.
Change map:
| Part | Result |
|---|---|
| Shared property redactor and registry | Master-equivalent name, JDBC,
Hadoop and address rules; all sensitive aliases declared by branch-4.2 property
holders are included |
| Metastore/storage annotations | Mark missing Glue/DLF/Iceberg/Paimon
access keys and S3/COS/OBS/OSS-HDFS session material sensitive without changing
binding |
| Catalog/audit/lineage hooks | Apply one policy to the supported branch-4.2
output paths, including both ALTER CATALOG audit hooks |
| Streaming S3 audit | Snapshot selected files and properties, apply the
current policy at emission, preserve native filenames, release snapshots after
audit |
| SQL replacement prerequisite | Reuse master’s codepoint-aware shared
helper so supplementary characters cannot leave credential suffixes outside the
replaced range |
| Config validation and forwarding | Reject invalid regex values before
publication; preserve backslashes and quotes when forwarding configuration SQL |
| Tests | Cover visible endpoints, secret values, disabled supplemental
rules, malformed/signed addresses, raw-value preservation and streaming task
lifecycle |
#### Masking rules
The display marker is `*XXX`. This is output redaction, not encryption of
stored metadata.
| Input | Output rule |
|---|---|
| Existing permanent secret names | Whole value masked, case-insensitively;
existing hidden properties remain omitted |
| MaxCompute | `mc.access_key`, `mc.secret_key`, `mc.session_token` masked |
| Ozone | `ozone.access_key`, `ozone.secret_key`, `ozone.session_token`
masked |
| Metastore credentials | All sensitive aliases from Glue, DLF, Iceberg
REST/JDBC, Paimon REST/JDBC and Lance REST holders are included; this covers
`aws.glue.session-token`, `iceberg.jdbc.password`, `paimon.jdbc.password`,
Paimon DLF access-key fields, and Lance bearer/API keys |
| Storage credentials | All existing sensitive
S3/GCS/Azure/OSS/OSS-HDFS/COS/OBS/MinIO/Ozone aliases remain protected;
session/security-token aliases are included. Retained `AZURE_ACCOUNT_KEY`,
`AZURE_CLIENT_SECRET`, `COS_ACCESS_KEY`, `COS_SECRET_KEY`, `COS_SESSION_TOKEN`,
`COS_TOKEN`, `OBS_ACCESS_KEY`, `OBS_SECRET_KEY`, `OBS_SESSION_TOKEN`,
`OBS_TOKEN`, `OSS_ACCESS_KEY`, `OSS_SECRET_KEY`, `OSS_SESSION_TOKEN`,
`OSS_TOKEN` and `dlf.catalog.securityToken` remain protected even when this
branch does not bind them |
| Explicit JDBC/connection-string names | Names ending in a delimited JDBC
URL/URI component, or
`connectionString`/`connection_string`/`connection-string`/`connection.string`,
mask the entire value, including `jdbc_url`, `jdbc.jdbc_url`,
`iceberg.jdbc.uri`, `paimon.jdbc.uri` |
| JDBC value under another name | Any trimmed value beginning with `jdbc:`
(case-insensitively) is entirely masked, including opaque Oracle syntax under
the shared `uri` alias |
| Native SDK credentials | Bare or delimited final `token`, `credential`,
`credentials` components mask the entire value |
| Trino passthrough | After `trino.`, any occurrence of `key`, `secret`,
`password`, `token`, `credential` triggers whole-value masking. Matching
follows master's case-insensitive substring semantics, including unusual
property names |
| Custom names | The case-insensitive supplemental regex is searched
anywhere in the property name; its default masks names containing `password`,
`passwd`, `secret`, `token`, `credential`, and common access/API-key spellings |
| Address names/values | Delimited final URI/URL/host/endpoint/address
names, including plurals, and recognized hierarchical URI-shaped values use the
address policy below |
Permanent Hadoop name families also mask whole values, with case-insensitive
matching and optional namespace prefixes before `fs.`:
| Family | Permanent credential names |
|---|---|
| `fs.s3.*`, `fs.s3a.*` | `access.key`, `secret.key`, `session.token`,
`encryption.key`, `server-side-encryption.key`, `proxy.password`; optional
`bucket.<bucket>.` scopes, including dotted bucket names |
| `fs.azure.*` | `account.key`, `sas`, `sas.fixed.token`,
`account.oauth2.client.secret`, `account.oauth2.refresh.token`,
`encryption.encoded.client-provided.key`, including account/container suffixes |
| `fs.adl.oauth2.*` | `credential`, `refresh.token` |
| `fs.oss.*` | `accessKeyId`, `accessKeySecret`, `securityToken` |
| `fs.cosn.*` | `secretId`, `secretKey`, including `userInfo.` scopes |
| `fs.obs.*` | `access.key`, `secret.key`, `session.token`, `security.token`
|
Address behavior:
- Ordinary host/port values and comma-separated plain host lists remain
visible, including bracketed IPv6. Numeric ports must be in the range 0–65535.
Long lists are checked iteratively.
- Supported schemes are `http`, `https`, `thrift`, `hdfs`, `viewfs`, `file`,
`s3`, `s3a`, `s3n`, `oss`, `cos`, `cosn`, `obs`, `gs`, `abfs`, `abfss`, `wasb`,
`wasbs`.
- Remove ordinary URL userinfo and fragments. Preserve safe authority/path
and non-secret query parameters. Decode parameter names before classification;
mask every occurrence of password/token/credential and
access/API/account/private/auth/application-key parameters, authorization, bare
`auth` and bare `key`. Supplemental name rules apply to query names too.
- A parameter containing `signature`, or a delimited `sig`/`sas`, masks the
entire URL. Unknown schemes, malformed authorities/ports/escapes/query names,
opaque formats, URL lists, ambiguous matrix paths and nested/over-encoded
connection strings also mask the entire value. No parser-error fallback prints
the raw input.
- Ordinary URL query separators are normalized to `&`. Repeated secret
parameters are all masked. Non-sensitive names containing nested URL/JDBC
values cause whole-address masking.
- Preserve ABFS/WASB container identity in `container@host`; it is native
storage syntax. Validate the container, rejecting encoded or password-bearing
forms. Preserve valid GCS bucket authorities with underscores and valid
HDFS/ViewFS default-authority locations and logical mount names.
- Unambiguous absolute local `driver_url` paths and absolute
`file:/...`/`file:///...` locations remain visible. Ambiguous local path syntax
is masked. The name-based secret rule always takes precedence.
- Only internal, already selected S3 file statuses enable native-file audit
handling. Sanitize each selected singleton before assembling the existing
escaped glob. Preserve literal `#`, percent spellings, `+`, spaces, separators
and empty query-shaped components that identify different objects; inspect
bounded decoded views for nested credentials without emitting those views.
Generic SQL URI lists do not enable this mode. Signed or unsafe selected
locations mask the whole batch.
Configuration:
`datasource_sensitive_property_pattern` is FE-local and mutable, with
default:
```text
password|passwd|secret|token|credential|access[._-]?key|api[._-]?key
```
Known secrets and this regex are combined by union. `(?!)` disables only the
supplemental rule; an empty regex matches every name. Invalid patterns are
rejected during startup, custom-config loading and runtime updates before
replacing the previous value. Configure every serving FE consistently.
Forwarded SQL preserves regex backslashes and quotes under both ordinary SQL
mode and `NO_BACKSLASH_ESCAPES`.
Output scope:
- SHOW CATALOG, SHOW CREATE CATALOG and `catalogs()` share the value-aware
policy.
- CREATE/ALTER CATALOG audit SQL uses the same property renderer. Shared
replacement ranges follow parser codepoint positions, preserving masking
boundaries after supplementary characters. Existing masking-enabled property
SQL paths, including resource/storage-vault and job audit formatting, inherit
the shared rules.
- Lineage omits name-sensitive properties and sanitizes remaining values.
- The branch-specific Lance SHOW CREATE TABLE path uses the same value
policy. Authentication-integration and Kafka specialized name checks retain
their existing additional heuristics.
- Successful, failed, notification-based and cancelled S3 streaming task
paths preserve the audit/lifecycle contract. Snapshots are released on the
worker after synchronous audit, including early cancellation.
Scope is structured property output. Arbitrary catalog comments, unrelated
free text/logging, raw resource proc output and external SDK containers without
a recognized secret name or supported address structure are not general-purpose
secret-scanned. Branch-4.2 has no master's provider plugin architecture or
typed diagnostic formatter; the port collects its existing sensitive
annotations directly and keeps bare `uri` value-aware rather than making every
URI globally secret. Stored properties, client inputs, replay, authentication
choices and artifact loading are unchanged.
### Testing
- Baseline RED: `PrintableMapTest#testMaxComputeSessionTokenIsMasked` ran
and failed with `expected mc.session_token = *XXX`, `actual mc.session_token =
fixture-session-material`.
- Unicode RED: the new supplementary-character audit test ran before the
shared-helper fix and failed; the malformed output retained the entire
`fixture-session-material` token. The final candidate includes this case and a
multi-range replacement case.
- Extended property/catalog/resource/offset verification: 742 tests across
70 selected classes, zero failures/errors, 10 existing disabled integration
cases skipped.
- Final focused verification after the Unicode prerequisite: 191 tests
across 10 classes, zero failures/errors/skips, including 24 property-policy
tests, catalog and streaming audit, config forwarding, parser, CREATE/ALTER
VIEW and async-MV dependencies. The corrected multi-range fixture was rerun in
its complete 24-test class; other focused cases use the identical production
code.
- FE Checkstyle: passed with the repository Maven validation.
- Local review: completed before the first commit. Fixed Trino name-matching
parity, the Unicode replacement prerequisite and test setup/API issues, then
repeated review until no new concrete findings remained. The full staged diff,
caller/range producers, lifecycle, raw-value preservation and submitted
metadata were reviewed.
- Extended `test_external_catalog_credential_masking` (`p0,nonConcurrent`)
covers SHOW/TVF output and persisted CREATE/ALTER audit SQL. Cluster regression
and real multi-FE RPC are not run locally.
Pre-submission status: Partially verified: cluster regression and real
multi-FE RPC are not run locally. Targeted FE verification, Checkstyle, staged
secret scanning and pre-commit review are complete with no open findings.
### Release note
External catalog display, audit property SQL and lineage metadata on
branch-4.2 now mask additional credentials and embedded connection secrets
consistently with master. Safe endpoints and supported native filesystem
identities remain readable. Runtime connection settings are unchanged.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [ ] No.
- [x] Yes. Additional structured credential values are hidden from
output.
- Does this need documentation?
- [ ] No.
- [x] Yes. Document the FE-local regex, fixed-rule union and structured
URL behavior in external catalog/security documentation.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]