Github user staple commented on a diff in the pull request:
https://github.com/apache/spark/pull/2491#discussion_r17866638
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -68,6 +74,54 @@ class NaiveBayesModel private[mllib] (
}
/**
+ * One block from a distributed model for a naive bayes classifier. The
model is divided into
+ * blocks, each containing the complete model state for a group of labels.
+ *
+ * @param labels array of labels
+ * @param pi log of class priors, with dimension C, the number of labels
in this block
+ * @param theta log of class conditional probabilities, with dimensions
C-by-D,
+ * where D is the number of features
+ */
+private case class NBModelBlock(labels: Array[Double], pi: BDV[Double],
theta: BDM[Double])
+
+/**
+ * Distributed model for a naive bayes classifier.
+ *
+ * @param modelBlocks RDD of NBModelBlock, comprising the model
+ */
+private class DistNaiveBayesModel(val modelBlocks: RDD[NBModelBlock])
extends NaiveBayesModel {
+
+ override def predict(testData: RDD[Vector]): RDD[Double] = {
+ // Pair each test data point with all model blocks.
+ val testXModel =
testData.map(_.toBreeze).zipWithIndex().cartesian(modelBlocks)
--- End diff --
Potentially the test vectors could be batched here to reduce the number of
intermediate distributed objects.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
