Harold,
just mentioning it in case you run into it: If you are in a separate
thread, there are apparently stricter limits to what you can and cannot
serialize:
val someVal
future {
// be very careful with defining RDD operations using someVal here
val myLocalVal = someVal
// use myLocalVal instead
}
On Thu, Oct 30, 2014 at 4:55 PM, Harold Nguyen <[email protected]> wrote:
> In Spark Streaming, when I do "foreachRDD" on my DStreams, I get a
> NonSerializable exception when I try to do something like:
>
> DStream.foreachRDD( rdd => {
> var sc.parallelize(Seq(("test", "blah")))
> })
>
Is this the code you are actually using? "var sc.parallelize(...)" doesn't
really look like valid Scala to me.
Tobias
