Re: Storing a JDBC-based table in a catalog for direct use in Spark SQL

2025-01-14 Thread Aaron Grubb
In case anyone comes across this, I managed to accomplish this using the Hive Standalone Metastore, with help from an article [1]. Required configurations: spark.sql.warehouse.dir=s3a:/// spark.sql.catalogImplementation=hive hive.metastore.uris=thrift://:9083 This enables me to run the example s

Storing a JDBC-based table in a catalog for direct use in Spark SQL

2025-01-13 Thread Aaron Grubb
Hi all, I'm trying to figure out how to persist a table definition in a catalog that can be used from different sessions. Something along the lines of --- CREATE TABLE spark_catalog.default.test_table ( name string ) USING jdbc OPTIONS ( driver 'com.mysql.cj.jdbc.Driver'