Thanx a lot for your reply. I can see kryo serialiser in the UI.
I have 1 another query :
I wanted to know the meaning of the following log message when running a
spark streaming job :
[spark-akka.actor.default-dispatcher-18] INFO
org.apache.spark.streaming.scheduler.JobScheduler - Total dela
nilmish,
To confirm your code is using kryo, go to the web ui of your application
(defaults to :4040) and look at the environment tab. If your serializer
settings are there then things should be working properly.
I'm not sure how to confirm that it works against typos in the setting, but
you can
static inner classes do not refer to the outer class. Often people
declare them non-static by default when it's unnecessary -- a
Comparator class is typically a great example. Anonymous inner classes
declared inside a method are another example, but there again they can
be refactored into named sta
I had issues around embedded functions here's what I have figured. Every
inner class actually contains a field referencing the outer class. The
anonymous class actually has a this$0 field referencing the outer class,
and thus why
Spark is trying to serialize Outer class.
In the Scala API, the clos
The error is resolved. I was using a comparator which was not serialised
because of which it was throwing the error.
I have now switched to kryo serializer as it is faster than java serialser.
I have set the required config
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerialize
I think Mayur meant that Spark doesn't necessarily clean the closure
under Java 7 -- is that true though? I didn't know of an issue there.
Some anonymous class in your (?) OptimisingSort class is getting
serialized, which may be fine and intentional, but it is not
serializable. You haven't posted
Hi Mayur, is that closure cleaning a JVM issue or a Spark issue? I'm used
to thinking of closure cleaner as something Spark built. Do you have
somewhere I can read more about this?
On Tue, Jun 3, 2014 at 12:47 PM, Mayur Rustagi
wrote:
> So are you using Java 7 or 8.
> 7 doesnt clean closures
So are you using Java 7 or 8.
7 doesnt clean closures properly. So you need to define a static class as a
function & then call that in your operations. Else it'll try to send the
whole class along with the function.
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi
Sorry if I'm dense but is OptimisingSort your class? it's saying you
have included something from it in a function that is shipped off to
remote workers but something in it is not java.io.Serializable.
OptimisingSort$6$1 needs to be Serializable.
On Tue, Jun 3, 2014 at 2:23 PM, nilmish wrote:
> I
