Thanks for the comments! Yes I agree we'll probably want to adjust the API, and we can definitely try to start with some reasonable placeholder around auth/secrets to unblock various parallel streams of work on federation while keeping an eye on how different secrets/auth models can be accommodated in the future without having to change the majority of the actual federation logic.
We can also mark this part of the API as "experimental - likely to have breaking changes" and gate any new in-development code behind a feature configuration parameter so any users don't accidentally depend on it before it's ready. I'm not sure if there's any way to label this in the OpenAPI spec since it's not the endpoint itself that's experimental, but just the specifying of the ConnectionConfigInfo -- in the meantime I guess I can at least add some text to the description of the field to indicate it's experimental. Any other suggestions for how best to mark a "work-in-progress API" are welcome. Also, to signal boost some context to others on this thread who may not have time to look at the PR in detail, I detailed some thoughts about how the secrets can be segregated safely under the hood regardless of where the secrets come from ( https://github.com/apache/polaris/pull/1026#discussion_r2004463034). The TL;DR is that even though the API objects show secrets directly in the config objects, under the hood the secrets will be decomposed into interactions with an internal SecretsManager interface to avoid storing plaintext secrets directly within the CatalogEntity. On Wed, Mar 19, 2025 at 3:53 PM Dmitri Bourlatchkov <di...@apache.org> wrote: > Thanks, Dennis, for starting this discussion. > > As for me the PR looks ok in its current state, although I posted a few > comments. > > I tend to think that in order to proceed with the federated catalogs > implementation, it is probably best to merge the PR now and later rework > the parts that deal with external connection authentication (i.e. use > Iceberg Auth Manager) and secret storage. > > However, that will likely lead to significant and non-backward-compatible > changes in Admin API areas that deal with federated catalogs. I am > personally fine with this approach. Still, I think it would be good to hear > what other people think about this. > > Cheers, > Dmitri. > > On Wed, Mar 19, 2025 at 6:08 PM Dennis Huo <huoi...@gmail.com> wrote: > > > Hi all, > > > > For anyone who may not have seen the PR on github, just wanted to ensure > > folks have a chance to give input on the mailing list here about the > latest > > API spec proposal for supporting Catalog Federation: > > > > https://github.com/apache/polaris/pull/1026 > > > > This adds a ConnectionConfigInfo field to Catalog with various subtypes > to > > represent different possible federation sources, but just beginning with > > "ICEBERG_REST" in the enum. This also defines how the authentication > > parameters for connecting to the remote catalog are defined or can be > > extended for other connection types. > > > > I know we have interest in adding other authentication modes soon for > > Iceberg REST and also adding other catalog sources in the near future > such > > as Hive Metastore, maybe Glue/S3Tables/OneLake/etc. > > > > The main tracking issue for Catalog Federation can be found here for more > > context: https://github.com/apache/polaris/issues/540 > > > > Any input much appreciated! > > >
