Jin-Hyeok, Cha created ZEPPELIN-4082:
----------------------------------------
Summary: Error occured when using UDF with scoped notebook
Key: ZEPPELIN-4082
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4082
Project: Zeppelin
Issue Type: Bug
Components: Interpreters
Affects Versions: 0.8.1
Environment: * Zeppelin v0.8.1
* Spark v2.4.0 (1 Master, N Workers)
* Hadoop (Embedded, Maybe v2.7.x)
* The interpreter will be instantiated *Per Note* in *scoped* process
Reporter: Jin-Hyeok, Cha
When I defined my own function with UDF (User-Defined Functions) feature, and I
got the error message like this:
{code:java}
java.lang.ClassCastException: cannot assign instance of
scala.collection.immutable.List$SerializationProxy to field
org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type
scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
{code}
I just defined the simple function:
{code:java}
import java.text.SimpleDateFormat
def diffHour(s1: String, s2: String): Long = {
var hour = 0L
try {
val sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
val d1 = sdf.parse(s1)
val d2 = sdf.parse(s2)
hour = d2.getTime - d1.getTime
hour /= 1000 * 60 * 60
} catch {
case e: Exception => hour = -1
}
hour
}{code}
And registered my function to Spark SQL Context:
{code:java}
sqlContext.udf.register("diffHour", diffHour _)
{code}
Now I expected I can use my function on SQL.
{code:java}
%sql
SELECT
id,
time,
diffHour(time, '2019-01-01 00:00:00') as hour
FROM users{code}
But the error occurred I mentioned at first.
I used *Per Note* and *scoped* settings for Spark Interpreter.
So I changed Interpreter settings to *Globally*.
Then error not occurred.
How can I fix it?
Please help me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
