Hi, Thank you for the feedback. I understand the concerns about adding more and more features to the protocol, especially if they might be implemented elsewhere. And every added bit of complexity should have clear cost/benefit ratio.
Iceberg is becoming the de-facto standard for multiple workloads around the world. In 2018 it was about huge companies. Now even relatively small businesses may benefit from it. Their infrastructure may differ dramaticallly: from fully-fledged clouds to small on-premises deployments. Thus lowering the bar for product adoption is important. Catalog is an integral part of Iceberg, even though we suppose that it is a third-party application. Unlike metadata layers in monolithic systems, catalog unavailability in lakehouses mean that all you data workload is effectively paralysed: no ETL from Spark, no ad-hoc analytics from Trino, no streaming ingestion from Kafka. Catalog becomes a huge single point of failure, so it becomes an important question - who is responsible for catalog HA now? It is true, that the problem can be solved elsewhere. E.g., one can set a load-balancer in front of catalog instances. But now you need HA for the load balancer itself, which often requires for additional infrastructure. So, yes, HA can be moved to another layer, but it doesn’t implies that this is a preferable approach. Hive Metastore is the most popular catalog in the world. It doesn’t think that HA is not its problem. At the client level you can provide multiple metastore URLs and also define the URL selection strategy ; (RANDOM, SEQUENTIAL). See [1]. This allows engines work with HA HMS without intermediaries. And from the client perspective the configuration is trivial. My proposal is to look for a simple yet sufficient solution at the Iceberg library level to allow for REST catalog HA without relying to other products. Looks like this could be done pretty much similarly to HMS client. The minimal required change is to allow multiple URLs for REST Catalog, and a relatively small change to the RESTSessionCatalog to switch between URLs in the case a special error is returned. In its simplest form it could be the receipt of 301 error code or so. This doesn’t seem like a serious complication, but will let REST catalog developers build implementations that are on par with HMS The n terms of HA. WDYT? [1] https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/hive-hms-ha-configuration.html *Vladimir Ozerov* Вт, 10 дек. 2024 г. в 00:57, Yufei Gu <flyrain...@gmail.com>: > Load balancing operates at a different layer than APIs, with various > implementations available, such as etcd and Zookeeper. I’d prefer to avoid > introducing additional complexity at the web service API level. > > > Yufei > > > On Mon, Dec 9, 2024 at 8:35 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Vladimir, >> >> As you said, today, it's possible to use a LB in front of multiple >> instances (using nginx, ELB, ...). >> I think it's pretty easy to setup and at "infrastructure" level. >> >> As it's possible to plug the HTTP5 client in Iceberg REST client, I >> think it's possible to inject PoolingHttpClientConnectionManager with >> multiple routes/host. >> >> I'm still prefer to define this at "infra" level (to decouple >> endpoints definition from the pure client). >> >> Regards >> JB >> >> >> On Mon, Dec 9, 2024 at 5:10 PM Vladimir Ozerov <voze...@querifylabs.com> >> wrote: >> > >> > Hi, >> > >> > Catalog is a critical part of Iceberg infrastructure and may require >> highly available setup. In similar services (e.g., HMS, etc) this is often >> done as follows: >> > >> > Start several service instances >> > Decide which one is coordinator via etcd, Zookeper, Ratis, etc >> > Expose HA endpoint to a client: multiple endpoints OR a single endpoint >> via proxy >> > >> > Currently, there is no way to expose multiple endpoints to a REST >> server. This may work in some cases if you hide multiple REST server >> instances behind a proxy/balancer. But this proxy requires own HA setup, >> which complicates the overall deployment. >> > >> > I'd like to ask the community whether we can extend REST specification >> with multiple endpoints to support HA REST catalog without proxies. This >> extension could contain two essential parts: >> > >> > How to provide multiple endpoints to RESTSessionCatalog. This could be >> encoded into URL or as additional property >> > Some additional headers and/or error codes to allow a REST server >> instance communicate peer endpoints and the current coordinator >> > >> > A very rough example how this can work in practice. Configuration: >> > >> > mycatalog.endpoint=https://host1 >> > mycatalog.endpoints=https://host1,https://host2,https://host3 >> > >> > REST server response headers with additional endpoints: >> > >> > X-HA-Coordinator: https://host2 >> > X-HA-Endpoints: https://host1,https://host2,https://host3 >> > >> > WDYT? >> > >> > Regards, >> > Vladimir >> >
