[
https://issues.apache.org/jira/browse/SPARK-55645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-55645.
------------------------------------
Fix Version/s: 4.2.0
Assignee: David Tagatac
Resolution: Fixed
Issue resolved by https://github.com/apache/spark/pull/54467
> Include Hive Metastore SerDeInfo field NAME in Spark representation
> -------------------------------------------------------------------
>
> Key: SPARK-55645
> URL: https://issues.apache.org/jira/browse/SPARK-55645
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.6
> Reporter: David Tagatac
> Assignee: David Tagatac
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> The [Hive Metastore
> API|https://github.com/apache/hive/blob/5160d3af392248255f68e41e1e0557eae4d95273/metastore/if/hive_metastore.thrift#L260]
> SerDeInfo struct includes a `name` field. However, this field is not
> included in the [data structure used to represent the API
> response|https://github.com/apache/spark/blob/ac4e7947e4efa4ca2484d9393b14322f4e636df5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L143-L149]
>
> {code:java}
> org.apache.spark.sql.catalyst.catalog.CatalogStorageFormat{code}
>
> As a result, extension developers do not have a good way to use this field
> without storing it somewhere unintuitive like
> `CatalogStorageFormat.properties` with some arbitrary key like "serde.name".
> Including this field as a corresponding field in the
> `org.apache.spark.sql.catalyst.catalog` data model will allow for a better
> representation of Hive Metastore tables in Spark.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
