Github user derrickburns commented on the pull request:
https://github.com/apache/spark/pull/2419#issuecomment-57023670
@mengxr I ran the tests according to your instructions. One issue remains
that I cannot resolve. *I need help with that issue*.
## Issues
First, the `org.apache.spark.mllib.clustering.KMeansClusterSuite` "task
size should be small in both training and prediction" fails, suggesting that
the RDD data is being captured in a closure. This is quite puzzling. My
efforts to solve this problem have failed. I *need help* to solve this problem.
Second, the` org.apache.spark.mllib.clustering.KMeansSuite` "two clusters"
unit test fails *intermittently* due to randomization. This is due to a bug
that exists in the current 1.1 release. This bug is NOT exacerbated by my pull
request. Nevertheless, I have modified my pull request to address this bug. An
explanation of the bug is below.
## Detail
### The data capture issue
Please look at line 139 of
`org.apache.spark.mllib.clustering.MultiKMeans.scala`. To avoid capture of the
data in the task closure, I have created an object with a single function.
That object does NOT capture the data. I then only reference that object in
the anonymous function. I do not understand how this line causes the data to
be captured in the closure of the anonymous function. I *need help* to solve
this. (I look forward to the acceptance of Scala SIP-21 ("spores") that appears
to address cleanly the issue of contaminated closures).
### The "two clusters" test failure
The random cluster center initialization algorithm uses sampling with
replacement. In the test case, there are 6 cluster centers to choose from and 2
are requested. In the Spark 1.1.0 implementation, cluster centers are chosen
randomly *with replacement*. Therefore, we can expect the test to fail 1/6 of
the time. The solution is to change the random initialization to sample
without replacement when possible. I have done is in a commit that I added to
this pull request.
### Another trivial bug fix
I found a bug in my new code where a toString method (that is only called
when an exception is thrown), would throw NPE exception itself. I fixed the
cause of the NPE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
