[
https://issues.apache.org/jira/browse/SPARK-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900825#comment-15900825
]
Nick Pentreath commented on SPARK-13969:
----------------------------------------
I think {{HashingTF}} and {{FeatureHasher}} are different things - similar to
HashingVectorizer and FeatureHasher in scikit-learn.
{{HashingTF}} (HashingVectorizer) transforms a Seq of terms (typically
{{String}}) into a term frequency vector. Yes technically it can operate on Seq
of any type (well actually only strings and numbers, see [murmur3Hash
function|https://github.com/apache/spark/blob/60022bfd65e4637efc0eb5f4cc0112289c783147/mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala#L151]).
It could certainly operate on multiple columns - that would hash all the
columns of sentences into one term frequency vector, so it seems like it would
probably be less used in practice (though Vowpal Wabbit supports a form of this
with its namespaces).
What {{HashingTF}} does not support is arbitrary categorical or numeric
columns. It is possible to support categorical "one-hot" style encoding using
what I have come to call the "stringify hack" - transforming a set of
categorical columns into a Seq for input to HashingTF.
So taking say two categorical columns {{city}} and {{state}}, for example:
{code}
+--------+-----+-------------------------+
|city |state|stringified |
+--------+-----+-------------------------+
|Boston |MA |[city=Boston, state=MA] |
|New York|NY |[city=New York, state=NY]|
+--------+-----+-------------------------+
{code}
This works but is pretty ugly, doesn't fit nicely into a pipeline, and can't
support numeric columns.
The {{FeatureHasher}} I propose acts like that in scikit-learn - it can handle
multiple numeric and/or categorical columns in one pass. I go into some detail
about all of this in my [Spark Summit East 2017
talk|https://www.slideshare.net/SparkSummit/feature-hashing-for-scalable-machine-learning-spark-summit-east-talk-by-nick-pentreath].
The rough draft of it used for the talk is
[here|https://github.com/MLnick/spark/blob/FeatureHasher/mllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala].
Another nice thing about the {{FeatureHasher}} is it opens up possibilities for
doing things like namespaces in Vowpal Wabbit and it would be interesting to
see if we could mimic their internal feature crossing, and so on.
> Extend input format that feature hashing can handle
> ---------------------------------------------------
>
> Key: SPARK-13969
> URL: https://issues.apache.org/jira/browse/SPARK-13969
> Project: Spark
> Issue Type: Sub-task
> Components: ML, MLlib
> Reporter: Nick Pentreath
> Priority: Minor
>
> Currently {{HashingTF}} works like {{CountVectorizer}} (the equivalent in
> scikit-learn is {{HashingVectorizer}}). That is, it works on a sequence of
> strings and computes term frequencies.
> The use cases for feature hashing extend to arbitrary feature values (binary,
> count or real-valued). For example, scikit-learn's {{FeatureHasher}} can
> accept a sequence of (feature_name, value) pairs (e.g. a map, list). In this
> way, feature hashing can operate as both "one-hot encoder" and "vector
> assembler" at the same time.
> Investigate adding a more generic feature hasher (that in turn can be used by
> {{HashingTF}}).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
