lasdf1234 commented on PR #14300: URL: https://github.com/apache/gravitino/pull/14300#issuecomment-6486592409
### Latest design (current tip) **Goal:** metalake / TMS lakehouse-iceberg catalogs keep working when a Gravitino-minted GCS OAuth2 token expires. This is not for Spark / external IRC HTTP clients. **Model** - `catalogCache` still holds `IcebergCatalogOperations` with the original idle eviction (`gravitino.catalog.cache.evictionIntervalMs`). It does **not** expire by GCS token. - Each lakehouse catalog owns its Iceberg client in `IcebergBackend` on the **lakehouse class loader**. - Hive / JDBC / REST all use the same path: `IcebergBackend.create` → own `IcebergCatalogWrapper` (+ helper / views). - Before use, if `gcs.oauth2.token-expires-at` is within **5 minutes**, remint via `withGcsServiceAccountCredentials` and rebuild the wrapper. - REST still builds Iceberg `RESTCatalog` and talks **HTTP** to the configured `uri` (e.g. IRC2). No borrow of auxiliary IRC wrappers, no in-process Jersey bridge, no `AuxiliaryIcebergCatalogBridge`. - IRC keeps its own `catalogWrapperCache` for Spark / external clients. Lakehouse does not share that cache or those wrapper instances. - `CatalogManager` / `GravitinoEnv` are unchanged. **Why this shape** - Sharing IRC wrappers across isolated class loaders is unsafe. - Borrowing IRC’s wrapper pinned a closed client after IRC’s token-based cache eviction. - Owning the client on the lakehouse side fixes GCS refresh for TMS the same way for Hive, JDBC, and REST. **Out of scope on this tip** - Process-in-process call into auxiliary IRC1 instead of HTTP for REST. - Expiring `catalogCache` by GCS token. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
