Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8561#discussion_r39005618
--- Diff: python/pyspark/ml/feature.py ---
@@ -167,6 +168,137 @@ def getSplits(self):
@inherit_doc
+class CountVectorizer(JavaEstimator, HasInputCol, HasOutputCol):
+ """
+ Extracts a vocabulary from document collections and generates a
[[CountVectorizerModel]].
+ >>> df = sqlContext.createDataFrame(
+ ... [(0, ["a", "b", "c"]), (1, ["a", "b", "b", "c", "a"])],
+ ... ["label", "raw"])
+ >>> cv = CountVectorizer(inputCol="raw", outputCol="vectors")
+ >>> model = cv.fit(df)
+ >>> model.transform(df).show(truncate=False)
+ +-----+---------------+-------------------------+
+ |label|raw |vectors |
+ +-----+---------------+-------------------------+
+ |0 |[a, b, c] |(3,[0,1,2],[1.0,1.0,1.0])|
+ |1 |[a, b, b, c, a]|(3,[0,1,2],[2.0,2.0,1.0])|
+ +-----+---------------+-------------------------+
+ ...
+ """
+
+ # a placeholder to make it appear in the generated doc
+ minTF = Param(
+ Params._dummy(), "minTF", "Filter to ignore rare words in" +
+ " a document. For each document, terms with frequency/count less
than the given" +
--- End diff --
Not part of this PR, but it would be nice to remove this boilerplate code.
There should be a way to document the params in `__iniit__()`. I'm thinking of
a decorator that can read `param.doc` and put it inside `__doc__`, but I'm not
sure whether it would work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
