Hi all, I would like to bring up a design discussion regarding HIVE-12679 and HIVE-27473, which aim to support custom catalogs (such as AWS Glue, Iceberg REST catalog, etc.).
According to the design document [1] authored by Okumin, there are two realistic approaches to support custom catalogs: - Option 1: Make IMetaStoreClient pluggable - Option 2: Make ThriftHiveMetastore.Iface pluggable Option 1 is the original approach proposed in HIVE-12679, and after extensive discussion in [2], we concluded that to make this viable, we need to refactor the existing IMetaStoreClient implementation to be composable. This effort is now being tracked under HIVE-27473, which I am currently working on. The draft implementation and design document can be found in [3] and [4], respectively. While Option 2 could lead to a simpler integration model for new clients, it might break existing integrations. On the other hand, if we can make Option 1 clean enough, it would support new catalog integrations and help decompose the complicated dependencies around the HiveMetaStoreClient family. Given these trade-offs, I would like to ask for community feedback before proceeding further. In particular: - Should we continue refining Option 1 under HIVE-27473 with the proposed architecture, aiming to preserve compatibility? - Or should we lean toward Option 2, favoring simplicity at the potential cost of compatibility? It would be greatly appreciated if anyone could share their insights and feedback on this. Thank you for your time and input. Regards, Seonggon Namgung [1] https://docs.google.com/document/d/1fFvB0DAXJvPYv27R8nLa3tOUT67-hY0owxBrDY-BxUA (Okumin, HIVE-12679) [2] https://github.com/apache/hive/pull/4444 [3] https://github.com/apache/hive/pull/5771 [4] https://docs.google.com/document/d/1lppPj35fBOOCjGNIouoJCsniMPpyzWx3zcE7kvu0Be4 (SG, HIVE-27473)
