GitHub user tmyklebu opened a pull request:
https://github.com/apache/spark/pull/593
[SPARK-1672][WIP] Separate partitioning in ALS
This is a work-in-progress. At present, the numbers of user and product
blocks are not exposed to the user and no tests are present. As such, this
should probably not be pulled as is.
This pull request changes the per-instance partitioner to two per-instance
partitioners, one for users and one for products.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tmyklebu/spark alsseppar
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/593.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #593
----
commit c774d7d4bff91c9387d059d1189799fa0ff1f4b0
Author: Tor Myklebust <[email protected]>
Date: 2014-04-14T22:01:18Z
Make the partitioner a member variable and use it instead of modding
directly.
commit c90b6d8e91f86cf89adf28de6f9185647c87e5c8
Author: Tor Myklebust <[email protected]>
Date: 2014-04-14T22:10:30Z
Scramble user and product ids before bucketing.
commit df27697649de50d364c42c76aebaebb34cbe87e2
Author: Tor Myklebust <[email protected]>
Date: 2014-04-15T19:47:17Z
Support custom partitioners. Currently we use the same partitioner for
users and products.
commit d872b098d41c4fc088e579e8fe199aca149bca64
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T12:19:48Z
Add negative id ALS test.
commit 36a0f43519a1e8ea800b960157f8c8b050139105
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T16:42:31Z
Make the partitioner private.
commit 5ec9e6cd237c4ac7c1b597614c880ae75bacceee
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T17:00:39Z
Clean a couple of things up using 'map'.
commit f8413451c807282100a9be506ca2c992abb81918
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T17:12:47Z
Fix daft bug creating 'pairs', also for -> foreach.
commit 40edc235e59aab56d6f65c73ffe98859c78a889b
Author: Tor Myklebust <[email protected]>
Date: 2014-04-16T18:14:38Z
Fix missing space.
commit 674933abb7a373dc1c913467d668bad9045e560f
Author: Tor Myklebust <[email protected]>
Date: 2014-04-19T23:36:52Z
Fix style.
commit 495784f2a172957ab490e0f77ea504c0179ab798
Author: Tor Myklebust <[email protected]>
Date: 2014-04-19T23:41:23Z
Merge branch 'master' of https://github.com/apache/spark
commit 23d6f91b52c88b7006ec78496f777b72b1881bb4
Author: Tor Myklebust <[email protected]>
Date: 2014-04-21T00:06:19Z
Stop making the partitioner configurable.
commit dcf583ac4001c6da8d6b85e45e88043861a351d8
Author: Tor Myklebust <[email protected]>
Date: 2014-04-21T19:56:48Z
Remove the partitioner member variable; instead, thread that needle
everywhere it needs to go.
commit 021f54b5d3d5177a5185442aaf3ddcde51a354a2
Author: Tor Myklebust <[email protected]>
Date: 2014-04-29T05:47:41Z
Separate user and product blocks.
commit bd8a75c53e1b84b4dfb493581bc1945eb93d624b
Author: Tor Myklebust <[email protected]>
Date: 2014-04-29T23:21:26Z
Merge branch 'master' of github.com:apache/spark into alsseppar
Conflicts:
mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
