[
https://issues.apache.org/jira/browse/SPARK-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826129#comment-15826129
]
Sean Owen commented on SPARK-14409:
-----------------------------------
BTW [~roberto.mirizzi] there are much simpler ways to write your match
statements with regexes if needed, and no reason to arbitrarily support only k
<= 10. We usually move to a pull request with [WIP] in the title if trying to
review substantial code but maybe we're not there yet.
What's the need for goodThreshold? All of the ranking metrics supported here
are a function of the top k predictions and top k "ground truth" relevant items
from a held-out set. Typically. I think that this is also implementable as
top-k per user query? but based on label not prediction.
This is probably a workable design to support precision and recall and MAP, but
I don't think it's a design that will support more general ranking metrics like
AUC. Hm, I haven't thought this through, but maybe the existing, separate
support for AUC would cover this case? I know it exists in MLlib.
I agree that this would have to be applied to the original data set, and not to
a subset as picked out by ALS. You are comparing the top-k items as predicted
by the model to the top-k items as defined by the label. I'm accustomed to
actually holding out those top-k from training too. I don't know how easy that
is to work into this design, and at some scale, it probably won't skew the
evaluation too much. But if the model is given all the answers including all
the top-k best items, then we're really just testing its ability to reconstruct
the input and a model that trivially returns answers based on the input data
directly would score perfectly.
> Investigate adding a RankingEvaluator to ML
> -------------------------------------------
>
> Key: SPARK-14409
> URL: https://issues.apache.org/jira/browse/SPARK-14409
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Reporter: Nick Pentreath
> Priority: Minor
>
> {{mllib.evaluation}} contains a {{RankingMetrics}} class, while there is no
> {{RankingEvaluator}} in {{ml.evaluation}}. Such an evaluator can be useful
> for recommendation evaluation (and can be useful in other settings
> potentially).
> Should be thought about in conjunction with adding the "recommendAll" methods
> in SPARK-13857, so that top-k ranking metrics can be used in cross-validators.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
