[
https://issues.apache.org/jira/browse/SPARK-19878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kavn qin updated SPARK-19878:
-----------------------------
Description:
When case class InsertIntoHiveTable intializes a serde it explicitly passes
null for the Configuration in Spark 1.5.0:
[https://github.com/apache/spark/blob/v1.5.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L58]
While in Spark 2.0.0, the HiveWriterContainer intializes a serde it also just
passes null for the Configuration:
[https://github.com/apache/spark/blob/v2.0.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala#L161]
When we implement a hive serde, we want to use the hive configuration to get
some static and dynamic settings, but we can not do it !
So this patch add the configuration when initialize hive serde.
was:
When case class InsertIntoHiveTable intializes a serde it explicitly passes
null for the Configuration in Spark 1.5.0:
[https://github.com/apache/spark/blob/v1.5.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L58]
While in Spark 2.0.0, the HiveWriterContainer intializes a serde it also just
passes null for the Configuration:
[https://github.com/apache/spark/blob/v2.0.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala#L161]
When we implement a hive serde, we want to use the hive configuration to get
some static and dynamic settings, we can not do it!
This patch just add the configuration when initialize hive serde.
> Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala
> ------------------------------------------------------------------------------
>
> Key: SPARK-19878
> URL: https://issues.apache.org/jira/browse/SPARK-19878
> Project: Spark
> Issue Type: Wish
> Components: SQL
> Affects Versions: 1.5.0, 1.6.0, 2.0.0
> Environment: Centos 6.5: Hadoop 2.6.0, Spark 1.5.0, Hive 1.1.0
> Reporter: kavn qin
> Priority: Minor
> Labels: patch
> Attachments: SPARK-19878.patch
>
>
> When case class InsertIntoHiveTable intializes a serde it explicitly passes
> null for the Configuration in Spark 1.5.0:
> [https://github.com/apache/spark/blob/v1.5.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L58]
> While in Spark 2.0.0, the HiveWriterContainer intializes a serde it also just
> passes null for the Configuration:
> [https://github.com/apache/spark/blob/v2.0.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala#L161]
> When we implement a hive serde, we want to use the hive configuration to get
> some static and dynamic settings, but we can not do it !
> So this patch add the configuration when initialize hive serde.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
