[
https://issues.apache.org/jira/browse/SPARK-19492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855996#comment-15855996
]
Sean Owen commented on SPARK-19492:
-----------------------------------
Hm. The thing is, this function by itself doesn't compile even in scala:
{code}
scala> val func = { case Department(_, name) => name == "hr" }
<console>:13: error: missing parameter type for expanded function
The argument types of an anonymous function must be fully known. (SLS 8.5)
Expected type was: ?
val func = { case Department(_, name) => name == "hr" }
{code}
but fine with an explicit type:
{code}
scala> val func: (Department => Boolean) = { case Department(_, name) => name
== "hr" }
func: Department => Boolean = $$Lambda$1328/543028700@6aa7b67f
{code}
The question is more why it works with a Seq at all. I don't know therefore
whether this is something that really should be expected to work like this or
whether it's just that Seq has some extra magic to infer this type of thing.
> Dataset, map, filter and pattern matching on elements
> -----------------------------------------------------
>
> Key: SPARK-19492
> URL: https://issues.apache.org/jira/browse/SPARK-19492
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2, 2.1.0
> Reporter: Loic Descotte
> Priority: Minor
>
> It seems it is impossible to use pattern matching to define input parameters
> for functions like filter, map, etc. on datasets.
> Example :
> This one is working :
> {code}
> val departments = Seq(
> Department(1, "hr"),
> Department(2, "it")
> ).toDS
> departments.filter{ d=>
> d.name == "hr"
> }
> {code}
> but not this one :
> {code}
> departments.filter{ case Department(_, name)=>
> name == "hr"
> }
> {code}
> Error :
> {code}
> error: missing parameter type for expanded function
> The argument types of an anonymous function must be fully known. (SLS 8.5)
> Expected type was: ?
> departments.filter{ case Department(_, name)=>
> {code}
> This kind of pattern matching should work (as departements dataset type is
> known) like Scala collections filter function, or RDD filter function for
> example.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
