GitHub user tmyklebu opened a pull request:
https://github.com/apache/spark/pull/568
Micro-optimisation of ALS
This change replaces some Scala `for` and `foreach` constructs with `while`
constructs. There may be a slight performance gain on the order of 1-2% when
training an ALS model.
I trained an ALS model on the Movielens 10M-rating dataset repeatedly both
with and without these changes. All 7 runs in both columns were done in a
Scala `for` loop like this:
for (iter <- 0 to 10) {
val before = System.currentTimeMillis()
val model = ALS.train(rats, 20, 10)
val after = System.currentTimeMillis()
println("%d ms".format(after-before))
println("rmse %g".format(computeRmse(model, rats, numRatings)))
}
The timings were done on a multiuser machine, and I stopped one set of
timings after 7 had been completed. It would be nice if somebody with
dedicated hardware could confirm my timings.
After Before
121980 ms 122041 ms
117069 ms 117127 ms
115332 ms 117523 ms
115381 ms 117402 ms
114635 ms 116550 ms
114140 ms 114076 ms
112993 ms 117200 ms
Ratios are about 1.0005, 1.0005, 1.019, 1.0175, 1.01671, 0.99944, and
1.03723. I therefore suspect these changes make for a slight performance gain
on the order of 1-2%.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tmyklebu/spark alsopt
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/568.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #568
----
commit c774d7d4bff91c9387d059d1189799fa0ff1f4b0
Author: Tor Myklebust <[email protected]>
Date: 2014-04-14T22:01:18Z
Make the partitioner a member variable and use it instead of modding
directly.
commit c90b6d8e91f86cf89adf28de6f9185647c87e5c8
Author: Tor Myklebust <[email protected]>
Date: 2014-04-14T22:10:30Z
Scramble user and product ids before bucketing.
commit df27697649de50d364c42c76aebaebb34cbe87e2
Author: Tor Myklebust <[email protected]>
Date: 2014-04-15T19:47:17Z
Support custom partitioners. Currently we use the same partitioner for
users and products.
commit d872b098d41c4fc088e579e8fe199aca149bca64
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T12:19:48Z
Add negative id ALS test.
commit 36a0f43519a1e8ea800b960157f8c8b050139105
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T16:42:31Z
Make the partitioner private.
commit 5ec9e6cd237c4ac7c1b597614c880ae75bacceee
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T17:00:39Z
Clean a couple of things up using 'map'.
commit f8413451c807282100a9be506ca2c992abb81918
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T17:12:47Z
Fix daft bug creating 'pairs', also for -> foreach.
commit 40edc235e59aab56d6f65c73ffe98859c78a889b
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T18:14:38Z
Fix missing space.
commit 674933abb7a373dc1c913467d668bad9045e560f
Author: Tor Myklebust <[email protected]>
Date: 2014-04-19T23:36:52Z
Fix style.
commit 495784f2a172957ab490e0f77ea504c0179ab798
Author: Tor Myklebust <[email protected]>
Date: 2014-04-19T23:41:23Z
Merge branch 'master' of https://github.com/apache/spark
commit 23d6f91b52c88b7006ec78496f777b72b1881bb4
Author: Tor Myklebust <[email protected]>
Date: 2014-04-21T00:06:19Z
Stop making the partitioner configurable.
commit dcf583ac4001c6da8d6b85e45e88043861a351d8
Author: Tor Myklebust <[email protected]>
Date: 2014-04-21T19:56:48Z
Remove the partitioner member variable; instead, thread that needle
everywhere it needs to go.
commit 114fb740c88da23372d2aaf6a12640d17b1408a1
Author: Tor Myklebust <[email protected]>
Date: 2014-04-25T23:19:33Z
Turn some 'for' loops into 'while' loops.
commit 4ef0313c80425f51fd42fe47a4f3ff4060471d03
Author: Tor Myklebust <[email protected]>
Date: 2014-04-26T19:11:52Z
Merge branch 'master' of github.com:apache/spark into alsopt
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
