[
https://issues.apache.org/jira/browse/SPARK-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475156#comment-15475156
]
Ryan Blue commented on SPARK-17302:
-----------------------------------
In 1.6.x, Spark pulled session config for Hive from a {{HiveConf}}, for example
[Spark
respects|https://github.com/apache/spark/blob/v1.6.1/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L167]
{{hive.exec.dynamic.partition.mode}}. This could be set in hive-conf.xml or in
spark-defaults.conf, though the latter had to use {{spark.hadoop...}} to set
it. Now that the {{SQLConf}} is used instead of {{HiveConf}}, you lose both of
those methods of setting Hive configuration variables. Using
{{spark.conf.set("key", "value")}} works for users, but hive-site.xml defaults
are no longer respected and an administrator can no longer set default values.
> Cannot set non-Spark SQL session variables in hive-site.xml,
> spark-defaults.conf, or using --conf
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-17302
> URL: https://issues.apache.org/jira/browse/SPARK-17302
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Ryan Blue
>
> When configuration changed for 2.0 to the new SparkSession structure, Spark
> stopped using Hive's internal HiveConf for session state and now uses
> HiveSessionState and an associated SQLConf. Now, session options like
> hive.exec.compress.output and hive.exec.dynamic.partition.mode are pulled
> from this SQLConf. This doesn't include session properties from hive-site.xml
> (including hive.exec.compress.output), and no longer contains Spark-specific
> overrides from spark-defaults.conf that used the spark.hadoop.hive... pattern.
> Also, setting these variables on the command-line no longer works because
> settings must start with "spark.".
> Is there a recommended way to set Hive session properties?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
