[
https://issues.apache.org/jira/browse/SPARK-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-19218:
----------------------------------
Description:
This issue aims to fix the following two things.
1. `sql("SET -v").collect()` or `sql("SET -v").show()` raises the following
exceptions for String configuration with default value, `null`. For the test,
please see [Jenkins
result](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71539/testReport/)
and
https://github.com/apache/spark/commit/60953bf1f1ba144e709fdae3903a390ff9479fd0
in #16624 .
{code}
sbt.ForkMain$ForkError: java.lang.RuntimeException: Error while decoding:
java.lang.NullPointerException
createexternalrow(input[0, string, false].toString, input[1, string,
false].toString, input[2, string, false].toString,
StructField(key,StringType,false), StructField(value,StringType,false),
StructField(meaning,StringType,false))
:- input[0, string, false].toString
: +- input[0, string, false]
:- input[1, string, false].toString
: +- input[1, string, false]
+- input[2, string, false].toString
+- input[2, string, false]
{code}
2. Currently, `SET` and `SET -v` commands show unsorted result.
We had better show a sorted result for UX. Also, this is compatible with
Hive.
*BEFORE*
{code}
scala> sql("set").show(false)
...
|spark.driver.host |10.22.16.140
|
|spark.driver.port |63893
|
|spark.repl.class.uri |spark://10.22.16.140:63893/classes
|
...
|spark.app.name |Spark shell
|
|spark.driver.memory |4G
|
|spark.executor.id |driver
|
|spark.submit.deployMode |client
|
|spark.master |local[*]
|
|spark.home |/Users/dhyun/spark
|
|spark.sql.catalogImplementation|hive
|
|spark.app.id |local-1484333618945
|
{code}
*AFTER*
{code}
scala> sql("set").show(false)
...
|spark.app.id |local-1484333925649
|
|spark.app.name |Spark shell
|
|spark.driver.host |10.22.16.140
|
|spark.driver.memory |4G
|
|spark.driver.port |64994
|
|spark.executor.id |driver
|
|spark.jars |
|
|spark.master |local[*]
|
|spark.repl.class.uri |spark://10.22.16.140:64994/classes
|
|spark.sql.catalogImplementation|hive
|
|spark.submit.deployMode |client
|
{code}
was:
Currently, `SET` command shows unsorted result. We had better show a sorted
result for UX. Also, this is compatible with Hive.
*BEFORE*
{code}
scala> sql("set").show(false)
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
|key |value
|
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
|spark.driver.host |10.22.16.140
|
|spark.driver.port |63893
|
|hive.metastore.warehouse.dir |file:/Users/dhyun/spark/spark-warehouse
|
|spark.repl.class.uri |spark://10.22.16.140:63893/classes
|
|spark.jars |
|
|spark.repl.class.outputDir
|/private/var/folders/bl/67vhzgqs1ks88l92h8dy8_1r0000gp/T/spark-43da424e-7530-4053-b30e-4068e8424dc9/repl-f1c957c7-2e4a-4f14-b234-f7b9f2447971|
|spark.app.name |Spark shell
|
|spark.driver.memory |4G
|
|spark.executor.id |driver
|
|spark.submit.deployMode |client
|
|spark.master |local[*]
|
|spark.home |/Users/dhyun/spark
|
|spark.sql.catalogImplementation|hive
|
|spark.app.id |local-1484333618945
|
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
{code}
*AFTER*
{code}
scala> sql("set").show(false)
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
|key |value
|
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
|hive.metastore.warehouse.dir
|file:/Users/dhyun/SPARK-SORTED-SET/spark-warehouse
|
|spark.app.id |local-1484333925649
|
|spark.app.name |Spark shell
|
|spark.driver.host |10.22.16.140
|
|spark.driver.memory |4G
|
|spark.driver.port |64994
|
|spark.executor.id |driver
|
|spark.home |/Users/dhyun/SPARK-SORTED-SET
|
|spark.jars |
|
|spark.master |local[*]
|
|spark.repl.class.outputDir
|/private/var/folders/bl/67vhzgqs1ks88l92h8dy8_1r0000gp/T/spark-ca1cf43c-ebd3-4e14-8045-0dd92419ef32/repl-006cff8e-fb82-4f18-9485-9fa9f91a2051|
|spark.repl.class.uri |spark://10.22.16.140:64994/classes
|
|spark.sql.catalogImplementation|hive
|
|spark.submit.deployMode |client
|
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+
{code}
> Fix SET command to show a result correctly and in a sorted order
> ----------------------------------------------------------------
>
> Key: SPARK-19218
> URL: https://issues.apache.org/jira/browse/SPARK-19218
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Dongjoon Hyun
> Priority: Trivial
>
> This issue aims to fix the following two things.
> 1. `sql("SET -v").collect()` or `sql("SET -v").show()` raises the following
> exceptions for String configuration with default value, `null`. For the test,
> please see [Jenkins
> result](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71539/testReport/)
> and
> https://github.com/apache/spark/commit/60953bf1f1ba144e709fdae3903a390ff9479fd0
> in #16624 .
> {code}
> sbt.ForkMain$ForkError: java.lang.RuntimeException: Error while decoding:
> java.lang.NullPointerException
> createexternalrow(input[0, string, false].toString, input[1, string,
> false].toString, input[2, string, false].toString,
> StructField(key,StringType,false), StructField(value,StringType,false),
> StructField(meaning,StringType,false))
> :- input[0, string, false].toString
> : +- input[0, string, false]
> :- input[1, string, false].toString
> : +- input[1, string, false]
> +- input[2, string, false].toString
> +- input[2, string, false]
> {code}
> 2. Currently, `SET` and `SET -v` commands show unsorted result.
> We had better show a sorted result for UX. Also, this is compatible with
> Hive.
> *BEFORE*
> {code}
> scala> sql("set").show(false)
> ...
> |spark.driver.host |10.22.16.140
>
> |
> |spark.driver.port |63893
>
> |
> |spark.repl.class.uri |spark://10.22.16.140:63893/classes
>
> |
> ...
> |spark.app.name |Spark shell
>
> |
> |spark.driver.memory |4G
>
> |
> |spark.executor.id |driver
>
> |
> |spark.submit.deployMode |client
>
> |
> |spark.master |local[*]
>
> |
> |spark.home |/Users/dhyun/spark
>
> |
> |spark.sql.catalogImplementation|hive
>
> |
> |spark.app.id |local-1484333618945
>
> |
> {code}
> *AFTER*
> {code}
> scala> sql("set").show(false)
> ...
> |spark.app.id |local-1484333925649
>
> |
> |spark.app.name |Spark shell
>
> |
> |spark.driver.host |10.22.16.140
>
> |
> |spark.driver.memory |4G
>
> |
> |spark.driver.port |64994
>
> |
> |spark.executor.id |driver
>
> |
> |spark.jars |
>
> |
> |spark.master |local[*]
>
> |
> |spark.repl.class.uri |spark://10.22.16.140:64994/classes
>
> |
> |spark.sql.catalogImplementation|hive
>
> |
> |spark.submit.deployMode |client
>
> |
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
