Chen Lin created SPARK-21005:
--------------------------------
Summary: VectorIndexerModel does not prepare output column field
correctly
Key: SPARK-21005
URL: https://issues.apache.org/jira/browse/SPARK-21005
Project: Spark
Issue Type: Bug
Components: MLlib
Affects Versions: 2.1.1
Reporter: Chen Lin
>From my understanding through reading the documentation, VectorIndexer
>decides which features should be categorical based on the number of distinct
>values, where features with at most maxCategories are declared categorical.
>Meanwhile, those features which exceed maxCategories are declared continuous.
Currently, VectorIndexerModel works all right with a dataset which has empty
schema. However, when VectorIndexerModel is transforming on a dataset with
`ML_ATTR` metadata, it may not output the expected result. For example, a
feature with nominal attribute which has distinct values exceeding maxCategorie
will not be treated as a continuous feature as we expected but still a
categorical feature. Thus, it may cause all the tree-based algorithms (like
Decision Tree, Random Forest, GBDT, etc.) throw errors as "DecisionTree
requires maxBins (= $maxPossibleBins) to be at least as large as the number of
values in each categorical feature, but categorical feature $maxCategory has
$maxCategoriesPerFeature values. Considering remove this and other categorical
features with a large number of values, or add more training examples.".
Correct me if my understanding is wrong.
I will submit a PR soon to solve this issue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
