laxmanmamididd commented on PR #7464: URL: https://github.com/apache/datafusion-comet/pull/7464#issuecomment-6935019884
Thanks for the careful read, @andygrove. > Could the rule restrict eviction to the case where the older entry's expiry is known to have passed or the new one is known to be later, and leave the no-expiry case to the LRU bound? Yes, done in 6a3279e1a. `FileIoCacheKey::supersedes` now evicts only when both entries carry `s3.session-token-expires-at-ms` and the new one is later. Any other difference keeps both entries and leaves them to the 64-entry LRU bound, as the release does. That covers properties with no expiry, such as an OAuth token or a tenant id, and an expiry on one side only. Two sessions or tenants that share a catalog name no longer evict each other on every task. The cost is that a provider re-vended without an expiry is closed only when the LRU bound retires its entry. REST catalogs that vend S3 credentials send the expiry, so they still get prompt release. Changes: - `supersedes` and its doc comment, and the eviction paragraph in `s3-credential-provider-design.md`, now say the above. - `cache_keeps_entries_without_a_later_expiry` replaces `cache_evicts_when_only_one_entry_has_an_expiry`. It covers two tenants with different OAuth properties, a one-sided expiry, and the LRU bound still retiring the least recently used entry. - The eviction tests, `CometS3CredentialReleaseSuite` and the superseded-FileIO test in `CometS3CredentialBridgeSuite` now give re-vended properties a later expiry. Tested locally on Spark 4.0 and JDK 17: the `iceberg_common` Rust tests (27 passed), plus `CometS3CredentialReleaseSuite` and `CometS3CredentialBridgeSuite` against Minio (11 passed). #7363 is rebased onto this commit and merged with `main` to clear its conflict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
