Gabriel39 opened a new pull request, #70675: URL: https://github.com/apache/doris/pull/70675
### Background Iceberg uses table encryption managers to unwrap metadata keys, and manifests can contain plaintext data-file or delete-file keys. Doris caches table owners and parsed manifests across statements; the SDK can also cache decrypted manifest bytes. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: A cached encryption manager can reuse an unwrapped key after the next statement loses KMS access. Parsed manifests, SDK byte-cache hits and shared unclassified loads can likewise reuse key-bearing metadata without validating the current read. Bound encrypted table ownership to statement leases, isolate unclassified loads and retain only plaintext manifest values. Encrypted SQL admission remains closed. ### Details - **Change map:** About 200 lines of production changes in cache retention/scoped loading and encrypted IO views, 774 lines of tests, and 37 lines of documentation. - **Frequency:** When encrypted metadata is warmed and then read by another statement or concurrently loaded before classification. - **Root cause:** Catalog-scoped cache ownership outlives statement-scoped key authorization. The SDK manifest byte cache is keyed by path and cannot validate the current descriptor from cached decrypted bytes. - **Why existing tests missed it:** Earlier cache tests primarily used plaintext metadata, and the encrypted fixture did not enable the SDK byte cache. Existing cancellation tests assumed unclassified results were safe to share. - **Fix:** Add a value-retention predicate covering count/weight limits and publication variants; use independent load scopes; bypass decoded encrypted results and SDK manifest byte caching; preserve active table leases until their final release. - **Other paths and branches:** Covers data, position-delete and equality-delete descriptors, synchronous/future reads, direct/bulk publication, replacement, refresh, concurrent cancellation and catalog retirement. No protocol fields or SQL enablement change. - **Behavior change:** Cold unclassified table/manifest misses may perform independent IO before a plaintext result is published. Completed plaintext entries remain reusable. Key-bearing metadata is not retained across statements; deterministic JVM memory zeroization is not promised. ### Testing - Full connector cache, SPI and Iceberg suites: 2,178 tests, zero failures/errors, five existing skips; FE Checkstyle passed. - ASAN native encryption/delete/scan tests: 147 passed. New coverage includes eight concurrent readers across AES key sizes, independent decryptor state, wrong-key/AAD retries and cleanup after success, cancellation and failed initialization. - Observed failing baselines for encrypted table retention, unclassified concurrent manifest loads, and a missing manifest descriptor bypassing the SDK byte cache; all corresponding tests now pass. - clang-format 16, BE header hygiene, clang-tidy and staged-diff checks passed. ### Release note Encrypted Iceberg metadata no longer remains in shared table or manifest caches across statements. Encrypted SQL read admission is unchanged. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the connector encryption-read design note is updated in this PR. - Pre-submission status: Ready for cache and lifetime hardening; encrypted SQL transport and end-to-end enablement are separate work. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests, documentation and this description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
