Hello.
I have activated the file checkpointing for a DStream to unleach the
updateStateByKey.
My unit test worked with no problem but when I have integrated this in my
full stream I got this exception. :
java.io.NotSerializableException: DStream checkpointing has been enabled but
the DStreams with their functions are not serializable
Serialization stack:
at
org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:550)
at
org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:587)
at
org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:586)
at com.misterbell.shiva.StreamingApp$.main(StreamingApp.scala:196)
at com.misterbell.shiva.StreamingApp.main(StreamingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
But this exception is not very clear about what part of my stream is not
serializable.
I try to add
.set("spark.driver.extraJavaOptions","-Dsun.io.serialization.extendedDebugInfo=true")
.set("spark.executor.extraJavaOptions","-Dsun.io.serialization.extendedDebugInfo=true")
to my spark conf to have more information, but it changes nothing ( it
should )
So how can I find which function or part of my stream is not serializable?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Stream-are-not-serializable-tp25185.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
