[
https://issues.apache.org/jira/browse/SPARK-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826285#comment-15826285
]
Danilo Ascione commented on SPARK-14409:
----------------------------------------
[~mlnick] This is a snippet to illustrate how I have dealt with the "null"
problem:
{code}
val predictionAndLabels: DataFrame = dataset
.join(topAtk, Seq($(queryCol)), "outer") //outer join to deal with nulls
in "label" column
.withColumn("topAtk", coalesce(col("topAtk"), mapToEmptyArray_()))
//coalease to deal with nulls in "prediction" column
.select($(labelCol), "topAtk")
{code}
>From line 111 of
>[RankingEvaluator|https://github.com/apache/spark/pull/16618/files#diff-0345c4cb1878d3bb0d84297202fdc95f]
> (I opened a PR for better readability)
> Investigate adding a RankingEvaluator to ML
> -------------------------------------------
>
> Key: SPARK-14409
> URL: https://issues.apache.org/jira/browse/SPARK-14409
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Reporter: Nick Pentreath
> Priority: Minor
>
> {{mllib.evaluation}} contains a {{RankingMetrics}} class, while there is no
> {{RankingEvaluator}} in {{ml.evaluation}}. Such an evaluator can be useful
> for recommendation evaluation (and can be useful in other settings
> potentially).
> Should be thought about in conjunction with adding the "recommendAll" methods
> in SPARK-13857, so that top-k ranking metrics can be used in cross-validators.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
