Hi, Catalog is a critical part of Iceberg infrastructure and may require highly available setup. In similar services (e.g., HMS, etc) this is often done as follows:
1. Start several service instances 2. Decide which one is coordinator via etcd, Zookeper, Ratis, etc 3. Expose HA endpoint to a client: multiple endpoints OR a single endpoint via proxy Currently, there is no way to expose multiple endpoints to a REST server. This may work in some cases if you hide multiple REST server instances behind a proxy/balancer. But this proxy requires own HA setup, which complicates the overall deployment. I'd like to ask the community whether we can extend REST specification with multiple endpoints to support HA REST catalog without proxies. This extension could contain two essential parts: 1. How to provide multiple endpoints to RESTSessionCatalog. This could be encoded into URL or as additional property 2. Some additional headers and/or error codes to allow a REST server instance communicate peer endpoints and the current coordinator A very rough example how this can work in practice. Configuration: mycatalog.endpoint=https://host1 mycatalog.endpoints=https://host1,https://host2,https://host3 REST server response headers with additional endpoints: X-HA-Coordinator: https://host2 X-HA-Endpoints: https://host1,https://host2,https://host3 WDYT? Regards, Vladimir
