GitHub user mengxr opened a pull request:
https://github.com/apache/spark/pull/1427
[SPARK-2361][MLLIB] Use broadcast instead of serializing data directly into
task closure
We saw task serialization problems with large feature dimension, which
could be avoid if we don't serialize data directly into task but use broadcast
variables. This PR uses broadcast in both training and prediction and adds
tests to make sure the task size is small.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mengxr/spark broadcast-new
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1427.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1427
----
commit 010d07692f40d42486cc53a053173f7e7bd6d44a
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-15T17:34:58Z
modify NaiveBayesModel and GLM to use broadcast
commit e00c2daa95c36db509006398ebdcb88874c525df
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-15T20:27:08Z
use broadcast in GD, KMeans
commit 1928a5a6d0c989b0124ce025e3a7245627cc4bb0
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-15T21:02:46Z
add test for KMeans task size
commit 174cb157c208e359455872da1c270a8ed8ccf7d2
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-15T23:30:47Z
use local-cluster for test with a small akka.frameSize
commit e73d68edc2b93f41749d95434a651d0f02c7ab55
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-15T23:31:10Z
update tests for k-means
commit 02103ba017e22f7b9a288dc21635eca998b96068
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-15T23:31:31Z
remove print
commit bccab926c6d3a62e57c536de1ae344dbf21a35c5
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T01:58:16Z
add task size test to LBFGS
commit 380778cc09dc64a904032652aabbf787421f188f
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:02:11Z
update KMeans test
commit 28a841123b1eb02dc495d1dd422a11a31a6b5456
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:10:31Z
add test for NaiveBayes
commit e0a5cf2876ccef1ad0e6be2c805cfacee849468b
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:18:41Z
add task size test to GD
commit 9427bf0cd3df6fb9728462dbf1c909119ec9a972
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:31:34Z
add task size tests to linear methods
commit 9ebadcc60b49b8773e773bac3f578e2d15616b4a
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:39:45Z
add task size test to RowMatrix
commit b97c1848afd88ddf025b80903981aa4d8bbe3eca
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:53:32Z
minimal change to LBFGS
commit b9a1228a360f2a9a45136f8da4b51990406441ff
Author: Xiangrui Meng <[email protected]>
Date: 2014-07-16T02:58:28Z
style update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
