What version of spark you are using? I cannot reproduce your error:
scala> spark.version
res9: String = 2.1.1
scala> val dataset = Seq((0, "hello"), (1, "world")).toDF("id", "text")
dataset: org.apache.spark.sql.DataFrame = [id: int, text: string]
scala> import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.udf
// define a method in similar way like you did
scala> def len = udf { (data: String) => data.length > 0 }
len: org.apache.spark.sql.expressions.UserDefinedFunction
// use it
scala> dataset.select(len($"text").as('length)).show
+------+
|length|
+------+
| true|
| true|
+------+
Yong
________________________________
From: Pralabh Kumar <[email protected]>
Sent: Friday, June 16, 2017 12:19 AM
To: lk_spark
Cc: user.spark
Subject: Re: how to call udf with parameters
sample UDF
val getlength=udf((data:String)=>data.length())
data.select(getlength(data("col1")))
On Fri, Jun 16, 2017 at 9:21 AM, lk_spark
<[email protected]<mailto:[email protected]>> wrote:
hi,all
I define a udf with multiple parameters ,but I don't know how to call it
with DataFrame
UDF:
def ssplit2 = udf { (sentence: String, delNum: Boolean, delEn: Boolean,
minTermLen: Int) =>
val terms = HanLP.segment(sentence).asScala
.....
Call :
scala> val output = input.select(ssplit2($"text",true,true,2).as('words))
<console>:40: error: type mismatch;
found : Boolean(true)
required: org.apache.spark.sql.Column
val output = input.select(ssplit2($"text",true,true,2).as('words))
^
<console>:40: error: type mismatch;
found : Boolean(true)
required: org.apache.spark.sql.Column
val output = input.select(ssplit2($"text",true,true,2).as('words))
^
<console>:40: error: type mismatch;
found : Int(2)
required: org.apache.spark.sql.Column
val output = input.select(ssplit2($"text",true,true,2).as('words))
^
scala> val output =
input.select(ssplit2($"text",$"true",$"true",$"2").as('words))
org.apache.spark.sql.AnalysisException: cannot resolve '`true`' given input
columns: [id, text];;
'Project [UDF(text#6, 'true, 'true, '2) AS words#16]
+- Project [_1#2 AS id#5, _2#3 AS text#6]
+- LocalRelation [_1#2, _2#3]
I need help!!
2017-06-16
________________________________
lk_spark
