Hello! I wanted to continue discussion that had started during review of the initial Catalog Federation API spec ( https://github.com/apache/polaris/pull/1026) regarding how we want to handle secrets.
I drafted this document describing how secrets management fits in to Catalog Federation, while extrapolating some interfaces that can define Polaris management of "user secrets" in a general way beyond federation as well: https://docs.google.com/document/d/1JPNx5vL4vM8DqwRwnBIPiQxwN4MXOdGx4Ki0j7vgwSM/edit?tab=t.0 One aspect incorporated in the design is the idea of independent trust boundaries between each of: - Secrets Manager (e.g. Vault, KMS, etc) - Persistence Database (e.g. Postgres, MongoDB, etc) - Service runtime (e.g. JVM running in k8s in a VM in cloud) The gist of it is that regardless of how users specify secrets attached to Polaris objects (such as an ExternalCatalog that is a federation facade), we canonicalize an internal "UserSecretReference" to be used in internal persistence objects (and internal persistence objects never contain a member holding a plaintext secret). The extensible UserSecretReference allows things like layered security and encryption such that neither the external secrets manager nor the persistence database alone store plaintext secrets in case either one is compromised independently. This PR is an end-to-end working MVP of Catalog Federation that includes an in-memory implementation of the UserSecretsManager exercising the layered-security flow; it may be helpful to see the secrets management in the full context of the working feature to see how its pieces fit together: https://github.com/apache/polaris/pull/1305 Would love to hear everyone's thoughts! Feel free to also comment in the document directly. Cheers, Dennis
