Hi Samrat, Thanks for the FLIP!
Since this is the first proposal for adding a vendor-specific catalog library in Flink, I think maybe we should also externalize those catalog libraries similar to how we are externalizing connector libraries. It is likely that we might want to add catalogs for other vectors in the future. Externalizing those catalogs can make Flink development more scalable in the long term. It is mentioned in the FLIP that there will be two types of SdkHttpClient supported based on the catalog option http-client.type. Is http-client.type a public config for the GlueCatalog? If yes, can we add this config to the "Configurations" section and explain how users should choose the client type? Regards, Dong On Sat, Dec 3, 2022 at 12:31 PM Samrat Deb <decordea...@gmail.com> wrote: > Hi everyone, > > I would like to open a discussion[1] on providing GlueCatalog support > in Flink. > Currently, Flink offers 3 major types of catalog[2]. Out of which only > HiveCatalog is a persistent catalog backed by Hive Metastore. We would like > to introduce GlueCatalog in Flink offering another option for users which > will be persistent in nature. Aws Glue data catalog is a centralized data > catalog in AWS cloud that provides integrations with many different > connectors[3]. Flink GlueCatalog can use the features provided by glue and > create strong integration with other services in the cloud. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-277%3A+Native+GlueCatalog+Support+in+Flink > > [2] > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/catalogs/ > > [3] > > https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#data-catalog-intro > > [4] https://issues.apache.org/jira/browse/FLINK-29549 > > Bests > Samrat >
