Igor Berman created ZEPPELIN-4179:
-------------------------------------
Summary: SparkScala211Interpreter missing synchronization
Key: ZEPPELIN-4179
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4179
Project: Zeppelin
Issue Type: Bug
Reporter: Igor Berman
Hi
using new spark interpreter with zeppelin.spark.useNew : true
with scala 2.11 and spak 2.2 sometimes creates very cryptic errors
while disabling it eliminates the problem
I believe it's connected to the fact that scala interpretation is not
synchronized, while in OldSparkInterpreter there is
synchronized (this) {} in interpret method
the errors we are getting varies from :
{code:java}
java.io.InvalidClassException:
$line187494590728.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw;
local class incompatible:
stream classdesc serialVersionUID = 665869549059 8467942,
local class serialVersionUID = 5448801901630725440{code}
to
{code:java}
java.lang.ClassCastException: cannot assign instance of
scala.collection.immutable.List$SerializationProxy to field
org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type
scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD at
java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133){code}
to reproduce define several notebooks with only difference of name of function
suffix(1,2,3,...), run then sequentially:
spark paragraph that defines simple udf:
{code:java}
import spark.implicits._
def num_to_ip3(num: Long) = {
require(num >= 0 && num < (2L << 31), s"Invalid number to convert: ${num}")
val buf : StringBuilder = new StringBuilder()
var i = 24
var ip = num
while (i >= 0) {
val a = ip >> i
ip = a << i ^ ip
buf.append(a)
if (i > 0) {
buf.append(".")
}
i = i - 8
}
buf.toString()
}
spark.udf.register("num_to_ip3", num_to_ip3 _)
{code}
and in 2nd paragraph usage of the above udf
{code:java}
%sql
select num_to_ip3(1342342){code}
[~mixermt] [~lio...@taboola.com] fyi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
