The streaming job has been running ok in 1.2 and 1.3. After I upgraded to 1.4, I started seeing error as below. It appears that it fails in validate method in StreamingContext. Is there anything changed on 1.4.0 w.r.t DStream checkpointint?
Detailed error from driver: 15/07/15 18:00:39 ERROR yarn.ApplicationMaster: User class threw exception: *java.io.NotSerializableException: DStream checkpointing has been enabled but the DStreams with their functions are not serializable* Serialization stack: java.io.NotSerializableException: DStream checkpointing has been enabled but the DStreams with their functions are not serializable Serialization stack: at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:550) at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:587) at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:586) -- Chen Song
