Have you checked out SparkCatalog <https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java> in Apache Iceberg project? More docs at https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs
On Fri, Mar 24, 2023 at 12:36 PM Alex Cruise <a...@cluonflux.com> wrote: > Hey folks, please let me know this is more of a user@ post! > > I'm building a Spark connector for my company's data-lake-ish product, and > it looks like there's very little documentation about how to go about it. > > I found ExternalCatalog a few days ago and have been implementing one of > those, but it seems like DataSourceRegister / SupportsCatalogOptions is > another popular approach. I'm not sure offhand how they overlap/intersect > just yet. > > I've also noticed a few implementations that put some of their code in > org.apache.spark.* packages in addition to their own; presumably this isn't > by accident. Is this practice necessary to get around package-private > visibility or something? > > Thanks! > > -0xe1a > -- John Zhuge
