[
https://issues.apache.org/jira/browse/SPARK-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437598#comment-15437598
]
Herman van Hovell edited comment on SPARK-16998 at 8/25/16 8:27 PM:
--------------------------------------------------------------------
I still have a code generation PR lying around:
https://github.com/apache/spark/pull/13065 That should fix a lot of the
performance issues.
I could bring it up to date, if there are any takers.
was (Author: hvanhovell):
I still have a code generation PR lying around:
https://github.com/apache/spark/pull/13065
I could bring it up to date.
> select($"column1", explode($"column2")) is extremely slow
> ---------------------------------------------------------
>
> Key: SPARK-16998
> URL: https://issues.apache.org/jira/browse/SPARK-16998
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: TobiasP
>
> Using a Dataset containing 10.000 rows, each containing null and an array of
> 5.000 Ints, I observe the following performance (in local mode):
> {noformat}
> scala> time(ds.select(explode($"value")).sample(false, 0.0000001, 1).collect)
> 1.219052 seconds
>
> res9: Array[org.apache.spark.sql.Row] = Array([3761], [3766], [3196])
> scala> time(ds.select($"dummy", explode($"value")).sample(false, 0.0000001,
> 1).collect)
> 20.219447 seconds
>
> res5: Array[org.apache.spark.sql.Row] = Array([null,3761], [null,3766],
> [null,3196])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
