[
https://issues.apache.org/jira/browse/SPARK-53447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-53447.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: This is a pure
how-to/support question (JIRA type "Question"): "how to configure multiple hive
metastores in one spark session for cross-cluster." No bug is claimed, no
reproducer is given, and no code/behavior change is requested. Verified against
the fork: docs/sql-data-sources-hive-tables.md documents the single-metastore
session config
(spark.sql.hive.metastore.version/jars/jars.path/sharedPrefixes/barrierPrefixes)
and the DataSourceV2 multi-catalog mechanism exists in
sql/catalyst/.../connector/catalog/ (CatalogManager.scala, Catalogs.scala,
LookupCatalog.scala) as the documented, user-ap
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> How to configure multiple Hive Metastores in a single Spark session?
> --------------------------------------------------------------------
>
> Key: SPARK-53447
> URL: https://issues.apache.org/jira/browse/SPARK-53447
> Project: Spark
> Issue Type: Question
> Components: Connect, Security, SQL
> Affects Versions: 3.3.1
> Environment: Spark 3.3.1
> Reporter: zane.young
> Priority: Minor
>
> In some of our scenarios, we need to perform *cross-cluster queries* by
> reading from multiple HDFS clusters within the same Spark session.
> To achieve this, we would like to configure *multiple Hive Metastores* in
> Spark, so that tables from different clusters can be queried without
> restarting the session.
> However, when I tried to configure Spark with multiple Hive Metastore URIs,
> it did not take effect.
> In addition, if multiple HMS are supported, how should *Kerberos
> authentication* be configured for each metastore?
> *Question:*
> * How can we configure multiple Hive Metastores in a single Spark session
> (for example, one for Hive tables and another for Hudi tables)? If this is
> possible, how can we query data from these different metastores using Spark
> SQL catalogs？
> * Is there a recommended way to handle *Kerberos authentication* for
> multiple HMS?
> Looking forward to your suggestions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
