Gabriel39 opened a new pull request, #70675:
URL: https://github.com/apache/doris/pull/70675

   ### Background
   
   Iceberg uses table encryption managers to unwrap metadata keys, and 
manifests can contain plaintext data-file or delete-file keys. Doris caches 
table owners and parsed manifests across statements; the SDK can also cache 
decrypted manifest bytes.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary:
   
   A cached encryption manager can reuse an unwrapped key after the next 
statement loses KMS access. Parsed manifests, SDK byte-cache hits and shared 
unclassified loads can likewise reuse key-bearing metadata without validating 
the current read. Bound encrypted table ownership to statement leases, isolate 
unclassified loads and retain only plaintext manifest values. Encrypted SQL 
admission remains closed.
   
   ### Details
   
   - **Change map:** About 200 lines of production changes in cache 
retention/scoped loading and encrypted IO views, 774 lines of tests, and 37 
lines of documentation.
   - **Frequency:** When encrypted metadata is warmed and then read by another 
statement or concurrently loaded before classification.
   - **Root cause:** Catalog-scoped cache ownership outlives statement-scoped 
key authorization. The SDK manifest byte cache is keyed by path and cannot 
validate the current descriptor from cached decrypted bytes.
   - **Why existing tests missed it:** Earlier cache tests primarily used 
plaintext metadata, and the encrypted fixture did not enable the SDK byte 
cache. Existing cancellation tests assumed unclassified results were safe to 
share.
   - **Fix:** Add a value-retention predicate covering count/weight limits and 
publication variants; use independent load scopes; bypass decoded encrypted 
results and SDK manifest byte caching; preserve active table leases until their 
final release.
   - **Other paths and branches:** Covers data, position-delete and 
equality-delete descriptors, synchronous/future reads, direct/bulk publication, 
replacement, refresh, concurrent cancellation and catalog retirement. No 
protocol fields or SQL enablement change.
   - **Behavior change:** Cold unclassified table/manifest misses may perform 
independent IO before a plaintext result is published. Completed plaintext 
entries remain reusable. Key-bearing metadata is not retained across 
statements; deterministic JVM memory zeroization is not promised.
   
   ### Testing
   
   - Full connector cache, SPI and Iceberg suites: 2,178 tests, zero 
failures/errors, five existing skips; FE Checkstyle passed.
   - ASAN native encryption/delete/scan tests: 147 passed. New coverage 
includes eight concurrent readers across AES key sizes, independent decryptor 
state, wrong-key/AAD retries and cleanup after success, cancellation and failed 
initialization.
   - Observed failing baselines for encrypted table retention, unclassified 
concurrent manifest loads, and a missing manifest descriptor bypassing the SDK 
byte cache; all corresponding tests now pass.
   - clang-format 16, BE header hygiene, clang-tidy and staged-diff checks 
passed.
   
   ### Release note
   
   Encrypted Iceberg metadata no longer remains in shared table or manifest 
caches across statements. Encrypted SQL read admission is unchanged.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; the connector encryption-read design 
note is updated in this PR.
   - Pre-submission status: Ready for cache and lifetime hardening; encrypted 
SQL transport and end-to-end enablement are separate work.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Code, tests, documentation and this description.
   - Human review: Pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to