Gilad,
I have a simple UDF to convert date running successfully with Zeppelin
0.6.0 & Spark 1.6.1
###### UDF
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import java.text.SimpleDateFormat
val toDate = udf((t: String) =>{
val sdf = new SimpleDateFormat("MM/dd/yy HH:mm").parse(t)
new java.sql.Date(sdf.getTime())
})
#### Later in another note
val df3 = df.withColumn("DateTimeOpened2", toDate('DateTimeOpened))
Looking at your exception there is CNF, perhaps related to a generated
class.
Can you please try with 0.6.0?
Thanks,
Vinay
On Sun, Jul 17, 2016 at 8:09 AM, Gilad Landau <[email protected]>
wrote:
> Hi,
>
> I’m trying to run simple UDF functions from Zeppelin and fail.
>
> I’m using Zeppelin 0.5.6 and it’s not even working on the simple Tutorial
> example, even though it does work from shell:
>
>
>
> *This is what I run:*
>
> def sentiment(s:String) : String = {
>
> val positive = Array("like", "love", "good", "great", "happy", "cool",
> "the", "one", "that")
>
> val negative = Array("hate", "bad", "stupid", "is")
>
>
>
> var st = 0;
>
>
>
> val words = s.split(" ")
>
> positive.foreach(p =>
>
> words.foreach(w =>
>
> if(p==w) st = st+1
>
> )
>
> )
>
>
>
> negative.foreach(p=>
>
> words.foreach(w=>
>
> if(p==w) st = st-1
>
> )
>
> )
>
> if(st>0)
>
> "positivie"
>
> else if(st<0)
>
> "negative"
>
> else
>
> "neutral"
>
> }
>
>
>
> // Below line works only in spark 1.3.0.
>
> // For spark 1.1.x and spark 1.2.x,
>
> // use sqlc.registerFunction("sentiment", sentiment _) instead.
>
> sqlc.udf.register("sentiment", sentiment _)
>
>
>
> *This is the Error I get:*
>
>
>
> import scala.util.matching.Regex
>
> import org.apache.spark.sql.functions.udf
>
> lengthString: (element_path: String)Int
>
> sentiment: (s: String)String
>
> lengthString_udf: org.apache.spark.sql.UserDefinedFunction =
> UserDefinedFunction(<function1>,IntegerType,List(StringType))
>
> res177: org.apache.spark.sql.UserDefinedFunction =
> UserDefinedFunction(<function1>,StringType,List(StringType))
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 140.0 failed 4 times, most recent failure: Lost task 0.3 in stage
> 140.0 (TID 19712, hdn02): java.lang.ClassNotFoundException:
> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>
>
>
>
>
> Any idea what causes it and how to fix?
>
>
>
> Thank you!
>
>
>
