[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanbo Liang updated SPARK-21591:
--------------------------------
Description:
The Tungsten execution engine substantially improved the efficiency of memory
and CPU for Spark application. However, in MLlib we still not migrate the
internal computing workload from {{RDD}} to {{DataFrame}}.
There are lots of blocking issues, lack of {{treeAggregate}} on {{DataFrame}}
is one of them. {{treeAggregate}} is very important for MLlib algorithms, since
they do aggregate on {{Vector}} which may has millions of elements. As we all
know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by an order
of magnitude for lots of MLlib
algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}} API
and do the performance benchmark related issues. And I think other scenarios
except for MLlib will also benefit from this improvement if we get it done.
was:
The Tungsten execution engine substantially improved the efficiency of memory
and CPU for Spark application. However, in MLlib we still not migrate the
internal computing workload from {{RDD}} to {{DataFrame}}.
There are lots of blocking issues, lack of {{treeAggregate}} on {{DataFrame}}
is one of them. It's very important for MLlib algorithms, since they do
aggregate on {{Vector}} which may has millions of elements. As we all know,
{{RDD}} based {{treeAggregate}} reduces the aggregation time by an order of
magnitude for lots of MLlib
algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}} API
and do the performance benchmark related issues. And I think other scenarios
except for MLlib will also benefit from this improvement if we get it done.
> Implement treeAggregate on Dataset API
> --------------------------------------
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> There are lots of blocking issues, lack of {{treeAggregate}} on {{DataFrame}}
> is one of them. {{treeAggregate}} is very important for MLlib algorithms,
> since they do aggregate on {{Vector}} which may has millions of elements. As
> we all know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by
> an order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
