Hi, I am developing a streaming application using checkpointing on Spark 1.5.1
I have just run into a NotSerializableException because some of the state that my streaming functions need cannot be serialized. This state is only used in the driver process, it is the checkpointing that requires the serialization. So I am considering moving that state into a Scala "object" - i.e. global singleton that must be mutable to allow the state to be set at application start. I would prefer to be able to create immutable state and attach it to either the SparkContext or SparkStreamingContext but I can't find an api for that. Does anybody else think is a good idea? Is there a better way? Or would such an api be a useful enhancement to Spark? Thanks in advance Simon Research Developer Viavi Solutions
