Gabriel39 opened a new pull request, #70776:
URL: https://github.com/apache/doris/pull/70776

   ### What problem does this PR solve?
   
   Issue Number: None
   
   Related PR: #70705 (merged master implementation; manually adapted to 
branch-4.2), #70366 (only its shared codepoint-aware SQL replacement helper)
   
   Background:
   
   External catalog properties reach SHOW output, SQL audit rendering and 
lineage metadata. Branch-4.2 uses a static `PrintableMap` registry rather than 
master's connector provider registration. Its registry omits several typed 
credential aliases and only examines property names, allowing retained session 
tokens, JDBC connection strings and credentials embedded in otherwise ordinary 
URLs to appear in output.
   
   Problem Summary:
   
   Bring the structured property masking behavior of #70705 to branch-4.2 while 
preserving the branch's Java 8 API compatibility and its existing catalog 
implementations. For example, formatting 
`mc.session_token=fixture-session-material` with masking enabled previously 
printed the value; it now prints `mc.session_token = *XXX`. Ordinary REST 
endpoints remain readable, while embedded userinfo and secret parameters are 
removed or masked. Runtime connection values remain intact.
   
   Details:
   
   - **Frequency:** Always for an affected retained property when a covered 
output path renders it.
   - **Root cause:** The legacy exact-name registry misses credential aliases 
and has no value-aware address policy. ALTER CATALOG also lacks both the 
audit-encryption interface and the property-range parser visitor on this 
branch. Its shared SQL replacement helper additionally treats parser codepoint 
positions as UTF-16 indices, which can truncate a masked range when SQL 
contains supplementary characters.
   - **Why existing tests missed it:** `PrintableMapTest` checks the older 
exact-name list and basic formatting. The existing external-catalog audit 
regression covers Iceberg OAuth fields but not MaxCompute/Ozone, JDBC URLs, 
custom/Hadoop credentials or URL userinfo. Existing S3 streaming audit tests 
check the selected batch but lack credential-bearing and literal-path boundary 
cases.
   - **Fix:** Keep a permanent credential union, add the same supplemental name 
and structured-value rules as master, and route the branch's existing output 
paths through it. Add both missing ALTER CATALOG audit hooks. Preserve selected 
S3 object identities while masking each internal singleton before building its 
display glob.
   - **Other paths and branches:** Master is fixed by #70705; this PR targets 
branch-4.2 only. The previous branch comparison also identified gaps on 
branch-4.1; it is not changed here. The omissions originate in the legacy 
registry; a single introducing PR has not been established.
   
   Change map:
   
   | Part | Result |
   |---|---|
   | Shared property redactor and registry | Master-equivalent name, JDBC, 
Hadoop and address rules; all sensitive aliases declared by branch-4.2 property 
holders are included |
   | Metastore/storage annotations | Mark missing Glue/DLF/Iceberg/Paimon 
access keys and S3/COS/OBS/OSS-HDFS session material sensitive without changing 
binding |
   | Catalog/audit/lineage hooks | Apply one policy to the supported branch-4.2 
output paths, including both ALTER CATALOG audit hooks |
   | Streaming S3 audit | Snapshot selected files and properties, apply the 
current policy at emission, preserve native filenames, release snapshots after 
audit |
   | SQL replacement prerequisite | Reuse master’s codepoint-aware shared 
helper so supplementary characters cannot leave credential suffixes outside the 
replaced range |
   | Config validation and forwarding | Reject invalid regex values before 
publication; preserve backslashes and quotes when forwarding configuration SQL |
   | Tests | Cover visible endpoints, secret values, disabled supplemental 
rules, malformed/signed addresses, raw-value preservation and streaming task 
lifecycle |
   
   #### Masking rules
   
   The display marker is `*XXX`. This is output redaction, not encryption of 
stored metadata.
   
   | Input | Output rule |
   |---|---|
   | Existing permanent secret names | Whole value masked, case-insensitively; 
existing hidden properties remain omitted |
   | MaxCompute | `mc.access_key`, `mc.secret_key`, `mc.session_token` masked |
   | Ozone | `ozone.access_key`, `ozone.secret_key`, `ozone.session_token` 
masked |
   | Metastore credentials | All sensitive aliases from Glue, DLF, Iceberg 
REST/JDBC, Paimon REST/JDBC and Lance REST holders are included; this covers 
`aws.glue.session-token`, `iceberg.jdbc.password`, `paimon.jdbc.password`, 
Paimon DLF access-key fields, and Lance bearer/API keys |
   | Storage credentials | All existing sensitive 
S3/GCS/Azure/OSS/OSS-HDFS/COS/OBS/MinIO/Ozone aliases remain protected; 
session/security-token aliases are included. Retained `AZURE_ACCOUNT_KEY`, 
`AZURE_CLIENT_SECRET`, `COS_ACCESS_KEY`, `COS_SECRET_KEY`, `COS_SESSION_TOKEN`, 
`COS_TOKEN`, `OBS_ACCESS_KEY`, `OBS_SECRET_KEY`, `OBS_SESSION_TOKEN`, 
`OBS_TOKEN`, `OSS_ACCESS_KEY`, `OSS_SECRET_KEY`, `OSS_SESSION_TOKEN`, 
`OSS_TOKEN` and `dlf.catalog.securityToken` remain protected even when this 
branch does not bind them |
   | Explicit JDBC/connection-string names | Names ending in a delimited JDBC 
URL/URI component, or 
`connectionString`/`connection_string`/`connection-string`/`connection.string`, 
mask the entire value, including `jdbc_url`, `jdbc.jdbc_url`, 
`iceberg.jdbc.uri`, `paimon.jdbc.uri` |
   | JDBC value under another name | Any trimmed value beginning with `jdbc:` 
(case-insensitively) is entirely masked, including opaque Oracle syntax under 
the shared `uri` alias |
   | Native SDK credentials | Bare or delimited final `token`, `credential`, 
`credentials` components mask the entire value |
   | Trino passthrough | After `trino.`, any occurrence of `key`, `secret`, 
`password`, `token`, `credential` triggers whole-value masking. Matching 
follows master's case-insensitive substring semantics, including unusual 
property names |
   | Custom names | The case-insensitive supplemental regex is searched 
anywhere in the property name; its default masks names containing `password`, 
`passwd`, `secret`, `token`, `credential`, and common access/API-key spellings |
   | Address names/values | Delimited final URI/URL/host/endpoint/address 
names, including plurals, and recognized hierarchical URI-shaped values use the 
address policy below |
   
   Permanent Hadoop name families also mask whole values, with case-insensitive 
matching and optional namespace prefixes before `fs.`:
   
   | Family | Permanent credential names |
   |---|---|
   | `fs.s3.*`, `fs.s3a.*` | `access.key`, `secret.key`, `session.token`, 
`encryption.key`, `server-side-encryption.key`, `proxy.password`; optional 
`bucket.<bucket>.` scopes, including dotted bucket names |
   | `fs.azure.*` | `account.key`, `sas`, `sas.fixed.token`, 
`account.oauth2.client.secret`, `account.oauth2.refresh.token`, 
`encryption.encoded.client-provided.key`, including account/container suffixes |
   | `fs.adl.oauth2.*` | `credential`, `refresh.token` |
   | `fs.oss.*` | `accessKeyId`, `accessKeySecret`, `securityToken` |
   | `fs.cosn.*` | `secretId`, `secretKey`, including `userInfo.` scopes |
   | `fs.obs.*` | `access.key`, `secret.key`, `session.token`, `security.token` 
|
   
   Address behavior:
   
   - Ordinary host/port values and comma-separated plain host lists remain 
visible, including bracketed IPv6. Numeric ports must be in the range 0–65535. 
Long lists are checked iteratively.
   - Supported schemes are `http`, `https`, `thrift`, `hdfs`, `viewfs`, `file`, 
`s3`, `s3a`, `s3n`, `oss`, `cos`, `cosn`, `obs`, `gs`, `abfs`, `abfss`, `wasb`, 
`wasbs`.
   - Remove ordinary URL userinfo and fragments. Preserve safe authority/path 
and non-secret query parameters. Decode parameter names before classification; 
mask every occurrence of password/token/credential and 
access/API/account/private/auth/application-key parameters, authorization, bare 
`auth` and bare `key`. Supplemental name rules apply to query names too.
   - A parameter containing `signature`, or a delimited `sig`/`sas`, masks the 
entire URL. Unknown schemes, malformed authorities/ports/escapes/query names, 
opaque formats, URL lists, ambiguous matrix paths and nested/over-encoded 
connection strings also mask the entire value. No parser-error fallback prints 
the raw input.
   - Ordinary URL query separators are normalized to `&`. Repeated secret 
parameters are all masked. Non-sensitive names containing nested URL/JDBC 
values cause whole-address masking.
   - Preserve ABFS/WASB container identity in `container@host`; it is native 
storage syntax. Validate the container, rejecting encoded or password-bearing 
forms. Preserve valid GCS bucket authorities with underscores and valid 
HDFS/ViewFS default-authority locations and logical mount names.
   - Unambiguous absolute local `driver_url` paths and absolute 
`file:/...`/`file:///...` locations remain visible. Ambiguous local path syntax 
is masked. The name-based secret rule always takes precedence.
   - Only internal, already selected S3 file statuses enable native-file audit 
handling. Sanitize each selected singleton before assembling the existing 
escaped glob. Preserve literal `#`, percent spellings, `+`, spaces, separators 
and empty query-shaped components that identify different objects; inspect 
bounded decoded views for nested credentials without emitting those views. 
Generic SQL URI lists do not enable this mode. Signed or unsafe selected 
locations mask the whole batch.
   
   Configuration:
   
   `datasource_sensitive_property_pattern` is FE-local and mutable, with 
default:
   
   ```text
   password|passwd|secret|token|credential|access[._-]?key|api[._-]?key
   ```
   
   Known secrets and this regex are combined by union. `(?!)` disables only the 
supplemental rule; an empty regex matches every name. Invalid patterns are 
rejected during startup, custom-config loading and runtime updates before 
replacing the previous value. Configure every serving FE consistently. 
Forwarded SQL preserves regex backslashes and quotes under both ordinary SQL 
mode and `NO_BACKSLASH_ESCAPES`.
   
   Output scope:
   
   - SHOW CATALOG, SHOW CREATE CATALOG and `catalogs()` share the value-aware 
policy.
   - CREATE/ALTER CATALOG audit SQL uses the same property renderer. Shared 
replacement ranges follow parser codepoint positions, preserving masking 
boundaries after supplementary characters. Existing masking-enabled property 
SQL paths, including resource/storage-vault and job audit formatting, inherit 
the shared rules.
   - Lineage omits name-sensitive properties and sanitizes remaining values.
   - The branch-specific Lance SHOW CREATE TABLE path uses the same value 
policy. Authentication-integration and Kafka specialized name checks retain 
their existing additional heuristics.
   - Successful, failed, notification-based and cancelled S3 streaming task 
paths preserve the audit/lifecycle contract. Snapshots are released on the 
worker after synchronous audit, including early cancellation.
   
   Scope is structured property output. Arbitrary catalog comments, unrelated 
free text/logging, raw resource proc output and external SDK containers without 
a recognized secret name or supported address structure are not general-purpose 
secret-scanned. Branch-4.2 has no master's provider plugin architecture or 
typed diagnostic formatter; the port collects its existing sensitive 
annotations directly and keeps bare `uri` value-aware rather than making every 
URI globally secret. Stored properties, client inputs, replay, authentication 
choices and artifact loading are unchanged.
   
   ### Testing
   
   - Baseline RED: `PrintableMapTest#testMaxComputeSessionTokenIsMasked` ran 
and failed with `expected mc.session_token = *XXX`, `actual mc.session_token = 
fixture-session-material`.
   - Unicode RED: the new supplementary-character audit test ran before the 
shared-helper fix and failed; the malformed output retained the entire 
`fixture-session-material` token. The final candidate includes this case and a 
multi-range replacement case.
   - Extended property/catalog/resource/offset verification: 742 tests across 
70 selected classes, zero failures/errors, 10 existing disabled integration 
cases skipped.
   - Final focused verification after the Unicode prerequisite: 191 tests 
across 10 classes, zero failures/errors/skips, including 24 property-policy 
tests, catalog and streaming audit, config forwarding, parser, CREATE/ALTER 
VIEW and async-MV dependencies. The corrected multi-range fixture was rerun in 
its complete 24-test class; other focused cases use the identical production 
code.
   - FE Checkstyle: passed with the repository Maven validation.
   - Local review: completed before the first commit. Fixed Trino name-matching 
parity, the Unicode replacement prerequisite and test setup/API issues, then 
repeated review until no new concrete findings remained. The full staged diff, 
caller/range producers, lifecycle, raw-value preservation and submitted 
metadata were reviewed.
   - Extended `test_external_catalog_credential_masking` (`p0,nonConcurrent`) 
covers SHOW/TVF output and persisted CREATE/ALTER audit SQL. Cluster regression 
and real multi-FE RPC are not run locally.
   
   Pre-submission status: Partially verified: cluster regression and real 
multi-FE RPC are not run locally. Targeted FE verification, Checkstyle, staged 
secret scanning and pre-commit review are complete with no open findings.
   
   ### Release note
   
   External catalog display, audit property SQL and lineage metadata on 
branch-4.2 now mask additional credentials and embedded connection secrets 
consistently with master. Safe endpoints and supported native filesystem 
identities remain readable. Runtime connection settings are unchanged.
   
   ### Check List (For Author)
   
   - Test
       - [x] Regression test
       - [x] Unit Test
       - [ ] Manual test
   
   - Behavior changed:
       - [ ] No.
       - [x] Yes. Additional structured credential values are hidden from 
output.
   
   - Does this need documentation?
       - [ ] No.
       - [x] Yes. Document the FE-local regex, fixed-rule union and structured 
URL behavior in external catalog/security documentation.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to