RE: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

2016-08-08 Thread Simon Scott
But does the “notSer” object have to be serialized? The object is immutable by the definition of A, so the only thing that needs to be serialized is the (immutable) Int value? And Ints are serializable? Just thinking out loud Simon Scott Research Developer @ viavisolutions.com From: Hao Ren

Futures timeout exception in executor logs

2017-09-14 Thread Simon Scott
, the driver dies without any logged message. Any advice much appreciated Thanks Simon Scott Viavi Solutions

Associating user objects with SparkContext/SparkStreamingContext

2016-06-24 Thread Simon Scott
Hi, I am developing a streaming application using checkpointing on Spark 1.5.1 I have just run into a NotSerializableException because some of the state that my streaming functions need cannot be serialized. This state is only used in the driver process, it is the checkpointing that requires th

RE: Associating user objects with SparkContext/SparkStreamingContext

2016-06-27 Thread Simon Scott
solution closer to the code that causes the problem. Cheers Simon From: Evan Sparks [mailto:evan.spa...@gmail.com] Sent: 24 June 2016 16:12 To: Simon Scott Cc: dev@spark.apache.org Subject: Re: Associating user objects with SparkContext/SparkStreamingContext I would actually think about this the