So far the best way to overcome it is to install all needed Julia packages on every machine. This is not very convenient, but at least it's not a blocker and you need to install some Julia packages manually anyway (though I'm thinking of creating API for massive installation of packages on all Spark workers).
I'm not sure what you mean by "Julia-on-C++", though. On Tue, Sep 22, 2015 at 5:33 AM, edward zhang <[email protected]> wrote: > > hmm, I'm already very interested in the project like > Julia-on-(Spark/c++?), > and the ser/des issue is a big obstacle. > 在 2015年9月21日星期一 UTC+8下午9:30:05，Andrei Zh写道： >> >> Hi, >> >> not yet. I made some initial research regarding serialization of ASTs and >> reconstructing functions from them, but it seems quite a tricky procedure >> and I have very little time for this project now. I plan to come back to >> this issue around the beginning of the next month. >> >> On Mon, Sep 21, 2015 at 11:25 AM, edward zhang <[email protected]> >> wrote: >> >>> hi， dear， >>> have you already fixed this problem？ >>> >>> >>> 在 2015年8月14日星期五 UTC+8下午11:06:30，Andrei Zh写道： >>> >>>> >>>> Hi Jake, >>>> >>>> your example works because you don't leave Julia session. `foo` is >>>> defined in this session, so the the pair of module name and function name >>>> is enough to get function object. If you save serialized function (or just >>>> retype it byte by byte) , it won't work. Here's an example: >>>> >>>> Session #1: >>>> >>>> julia> io = IOBuffer() >>>> IOBuffer(data=Uint8[...], readable=true, writable=true, seekable=true, >>>> append=false, size=0, maxsize=Inf, ptr=1, mark=-1) >>>> >>>> >>>> julia> foo(x) = x + 1 >>>> foo (generic function with 1 method) >>>> >>>> >>>> julia> serialize(io, foo) >>>> >>>> >>>> julia> takebuf_array(io) >>>> 9-element Array{Uint8,1}: >>>> 0x13 >>>> 0x02 >>>> 0x23 >>>> 0x2f >>>> 0x02 >>>> 0x03 >>>> 0x66 >>>> 0x6f >>>> 0x6f >>>> >>>> >>>> julia> >>>> >>>> >>>> >>>> Session #2: >>>> >>>> julia> data = Uint8[0x13, 0x02, 0x23, 0x2f, 0x02, 0x03, 0x66, 0x6f, >>>> 0x6f] >>>> 9-element Array{Uint8,1}: >>>> 0x13 >>>> 0x02 >>>> 0x23 >>>> 0x2f >>>> 0x02 >>>> 0x03 >>>> 0x66 >>>> 0x6f >>>> 0x6f >>>> >>>> >>>> julia> io = IOBuffer(data) >>>> IOBuffer(data=Uint8[...], readable=true, writable=false, seekable=true, >>>> append=false, size=9, maxsize=Inf, ptr=1, mark=-1) >>>> >>>> >>>> julia> bar = deserialize(io) >>>> (anonymous function) >>>> >>>> >>>> julia> bar(1) >>>> ERROR: function foo not defined on process 1 >>>> in error at error.jl:21 >>>> in anonymous at serialize.jl:398 >>>> >>>> >>>> julia> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Friday, August 14, 2015 at 5:49:55 PM UTC+3, Jake Bolewski wrote: >>>>> >>>>> Andrei Zh >>>>> >>>>> I'm confused. Have you actually tried? >>>>> >>>>> julia> io = IOBuffer() >>>>> IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, >>>>> append=false, size=0, maxsize=Inf, ptr=1, mark=-1) >>>>> >>>>> julia> foo(x) = x + 1 >>>>> foo (generic function with 1 method) >>>>> >>>>> julia> serialize(io, foo) >>>>> >>>>> julia> seekstart(io) >>>>> IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, >>>>> append=false, size=9, maxsize=Inf, ptr=1, mark=-1) >>>>> >>>>> julia> baz = deserialize(io) >>>>> foo (generic function with 1 method) >>>>> >>>>> julia> baz(1) >>>>> 2 >>>>> >>>>> The serialization code won't recursively serialize all the of the >>>>> functions dependencies so you will have to send/serialize the code that >>>>> defines the environment (types, constants, Packages, etc). >>>>> >>>> >>
