Gabriel39 opened a new pull request, #70159:
URL: https://github.com/apache/doris/pull/70159
### What problem does this PR solve?
Related PR: #70087
Problem Summary:
Enabling Parquet modular encryption in Arrow does not implement native
encrypted Iceberg reads in Doris. Scan planning currently has no shared
admission check for an encrypted table. Ordinary scans can emit ranges without
establishing encryption support, while metadata-only COUNT and system-table
routes can bypass the native data reader altogether.
For example, a v3 table whose SDK metadata contains `encryption.key-id` can
reach scan planning and COUNT pushdown without any proof that the selected
backends understand encrypted Iceberg tasks. This PR establishes the
first-stage read contract before native execution is enabled.
- Add `enable_encryption_read` to the Iceberg plugin's `iceberg.conf`,
defaulting to `false`. This is deployment configuration requiring an FE
restart, with no catalog/session override or fe.conf fallback.
- Share admission checks across table metadata and snapshot binding,
ordinary scans, COUNT eligibility/pushdown, streaming scans, system tables and
explicitly empty snapshot pins.
- Detect encryption from the table key property, retained v3 encryption
keys, or a non-plaintext SDK manager. Removing the current write key must not
admit old encrypted snapshots as plaintext.
- Return stable classifications for disabled reads, unsupported
catalogs/table formats/custom managers, and unavailable backend capabilities.
Error text excludes key identifiers and encryption objects.
- Even with the switch on, reject encrypted execution until backend
capability negotiation and native readers exist. A declared encrypted table is
rejected before KMS manager initialization or manifest I/O. Plaintext scans and
exact metadata counts retain their behavior.
The included developer contract defines the initial HMS / v3 / native
Parquet `AES_GCM_V1` target and the requirements for explicit BE advertisement,
placement, retries, key ownership and redaction. This PR deliberately adds no
SPI/wire fields and grants no encrypted-write capability. Actual per-file
format/algorithm validation and undeclared externally encrypted files remain
part of the subsequent planner/reader work.
### Validation
```sh
JAVA_TOOL_OPTIONS='-XX:ActiveProcessorCount=8' mvn -f fe/pom.xml -pl
:fe-connector-iceberg -am package -Dmaven.build.cache.enabled=false
mvn -f fe/pom.xml -pl :fe-connector-iceberg checkstyle:check
```
- The complete reactor package succeeded, including connector packaging and
upstream module tests. Iceberg connector results: **1923 tests reported, 1909
passed, 14 existing skips, 0 failures, 0 errors**.
- Added 8 contract tests and 1 configuration test. Real SDK metadata
fixtures cover empty/populated tables, direct/cached acquisition, statement
scopes, metadata binding, COUNT, streaming, system tables, retained keys,
unsupported catalogs/v2/custom managers, error redaction and rejection before
KMS initialization or manifest/data I/O.
- Before the implementation, five initial contract tests failed because
encrypted reads were admitted; the plaintext compatibility test passed. The
final added contract/configuration tests all pass.
- FE Checkstyle: **0 violations**. Verified that the plugin zip includes the
admission class and the updated configuration template.
- Local HTTP mock servers were run without proxy interception. The test JVM
uses 8 processors so the existing cancellation fixtures do not trigger the
SDK's automatic manifest-merge threshold. No existing test was changed or
excluded for this.
Validation is at the connector/SDK metadata boundary. Full Doris SQL, live
KMS decryption and mixed-version cluster integration were not run; encrypted
scan execution remains unavailable at this stage.
### Release note
Declared encrypted Iceberg tables now fail with clear read-admission errors.
The plugin read switch defaults off, and enabling it cannot bypass unavailable
backend support or enable encrypted writes.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test
- Behavior changed:
- [x] Yes. Declared encrypted tables are rejected consistently before
scan execution or metadata-count shortcuts.
- Does this need documentation?
- [x] Yes. Developer contract and configuration comments are included in
this PR; public query documentation follows when encrypted execution is
implemented.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
