Hi Tobias, On Wed, May 21, 2014 at 5:45 PM, Tobias Pfeiffer <t...@preferred.jp> wrote: >first, thanks for your explanations regarding the jar files! No prob :-)
> On Thu, May 22, 2014 at 12:32 AM, Gerard Maas <gerard.m...@gmail.com> > wrote: > > I was discussing it with my fellow Sparkers here and I totally overlooked > > the fact that you need the class files to de-serialize the closures (or > > whatever) on the workers, so you always need the jar file delivered to > the > > workers in order for it to work. > > So the closure as a function is serialized, sent across the wire, > deserialized there, and *still* you need the class files? (I am not > sure I understand what is actually sent over the network then. Does > that serialization only contain the values that I close over?) > I also had that mental lapse. Serialization refers to converting object (not class) state (current values) into a byte stream and de-serialization restores the bytes from the wire into an seemingly identical object at the receiving side (except for transient variables), for that, it requires the class definition of that object to know what it needs to instantiate, so yes, the compiled classes need to be given to the Spark driver and it will take care of dispatching them to the workers (much better than in the old RMI days ;-) > If I understand correctly what you are saying, then the documentation > at https://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html > (list item 8) needs to be extended quite a bit, right? > The mesos docs have been recently updated here: https://github.com/apache/spark/pull/756/files Don't know where the latest version from master is built/available. -kr, Gerard.
