Michael Allman created SPARK-18507:
--------------------------------------
Summary: Major performance regression in SHOW PARTITIONS on
partitioned Hive tables
Key: SPARK-18507
URL: https://issues.apache.org/jira/browse/SPARK-18507
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.0
Reporter: Michael Allman
Priority: Critical
Commit {{ccb11543048dccd4cc590a8db1df1d9d5847d112}}
(https://github.com/apache/spark/commit/ccb11543048dccd4cc590a8db1df1d9d5847d112)
appears to have introduced a major regression in the performance of the Hive
{{SHOW PARTITIONS}} command. Running that command on a Hive table with 17,337
partitions in the {{spark-sql}} shell with the parent commit of {{ccb1154}}
takes approximately 7.3 seconds. Running the same command with commit
{{ccb1154}} takes approximately 250 seconds.
I have not had the opportunity to complete a thorough investigation, but I
suspect the problem lies in the diff hunk beginning at
https://github.com/apache/spark/commit/ccb11543048dccd4cc590a8db1df1d9d5847d112#diff-159191585e10542f013cb3a714f26075L675.
If that's the case, this performance issue should manifest itself in other
areas as this programming pattern was used elsewhere in this commit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
