Re: Adding new connectors

2023-03-27 Thread Russell Spitzer
External Catalog is old and you probably shouldn't touch it, it's mostly for wrapping the old Hive interface. Almost all connectors now are built using the DSV2 interface either via a catalog, or directly via DataSourceRegister. I think Datasource register is mostly good for backwards compatibilit

Re: Adding new connectors

2023-03-27 Thread Alex Cruise
On Fri, Mar 24, 2023 at 11:23 AM Alex Cruise wrote: > I found ExternalCatalog a few days ago and have been implementing one of > those, but it seems like DataSourceRegister / SupportsCatalogOptions is > another popular approach. I'm not sure offhand how they overlap/intersect > just yet. > I wou

Re: Adding new connectors

2023-03-24 Thread Alex Cruise
On Fri, Mar 24, 2023 at 3:18 PM John Zhuge wrote: > Is this similar to Iceberg's hidden partitioning > ? > Check out the details in the spec: > https://iceberg.apache.org/spec/#partition-transforms > Yes, it's ver

Re: Adding new connectors

2023-03-24 Thread John Zhuge
Is this similar to Iceberg's hidden partitioning ? Check out the details in the spec: https://iceberg.apache.org/spec/#partition-transforms On Fri, Mar 24, 2023 at 2:52 PM Alex Cruise wrote: > On Fri, Mar 24, 2023

Re: Adding new connectors

2023-03-24 Thread Alex Cruise
On Fri, Mar 24, 2023 at 1:46 PM John Zhuge wrote: > Have you checked out SparkCatalog > > in > Apache Iceberg project? More docs at > https://iceberg.apache.org/docs/latest/s

Re: Adding new connectors

2023-03-24 Thread John Zhuge
Have you checked out SparkCatalog in Apache Iceberg project? More docs at https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs On Fri, Mar 24, 2023 at 12:36 PM