Have you tried Dagger.jl <https://github.com/JuliaParallel/Dagger.jl> to set up a DAG of computations you need performed?
On Monday, September 12, 2016 at 5:04:26 PM UTC-4, Adrian Salceanu wrote: > > This is a random example of an error - not really sure how to debug this, > seems to crash within the Postgres library. The dump is long but not really > helpful. > > 12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 5 > 12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 3 > 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 2 > fatal error on fatal error on 3: ERROR: attempt to send to unknown socket > 4: ERROR: attempt to send to unknown socket > 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 > 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 5 > fatal error on 3: ERROR: attempt to send to unknown socket > 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 > fatal error on 2: ERROR: attempt to send to unknown socket > ERROR: LoadError: On worker 2: > LoadError: On worker 2: > LoadError: LoadError: LoadError: LoadError: LoadError: > ProcessExitedException() > in yieldto at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in wait at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib (repeats 3 > times) > in take! at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib (repeats > 2 times) > in remotecall_fetch at multi.jl:745 > in remotecall_fetch at multi.jl:750 > in anonymous at multi.jl:1396 > > ...and 1 other exceptions. > > in include_string at loading.jl:282 > in include_from_node1 at > /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in include_string at loading.jl:282 > in include_from_node1 at > /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in eval at > /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Database.jl:1 > in include_string at loading.jl:282 > in include_from_node1 at > /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in include_string at loading.jl:282 > in include_from_node1 at > /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in eval at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in anonymous at multi.jl:1394 > in run_work_thunk at multi.jl:661 > in remotecall_fetch at multi.jl:734 > in remotecall_fetch at multi.jl:750 > in anonymous at multi.jl:1396 > while loading /Users/adrian/.julia/v0.4/PostgreSQL/src/../deps/build.jl, > in expression starting on line 9 > while loading /Users/adrian/.julia/v0.4/PostgreSQL/src/PostgreSQL.jl, in > expression starting on line 8 > while loading > /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/database_adapters/PostgreSQLDatabaseAdapter.jl, > > in expression starting on line 3 > while loading > /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Database.jl, in > expression starting on line 7 > while loading > /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/commands.jl, in > expression starting on line 11 > in remotecall_fetch at multi.jl:735 > in remotecall_fetch at multi.jl:750 > in anonymous at multi.jl:1396 > > ...and 1 other exceptions. > > in sync_end at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in anonymous at multi.jl:1405 > in include_string at loading.jl:282 > in include_from_node1 at > /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in eval at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in anonymous at multi.jl:1394 > in anonymous at multi.jl:923 > in run_work_thunk at multi.jl:661 > [inlined code] from multi.jl:923 > in anonymous at task.jl:63 > while loading > /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Genie.jl, in expression > starting on line 1 > in remotecall_fetch at multi.jl:747 > in remotecall_fetch at multi.jl:750 > in anonymous at multi.jl:1396 > > ...and 4 other exceptions. > > in sync_end at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in anonymous at multi.jl:1405 > in include at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in include_from_node1 at > /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in process_options at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > in _start at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib > while loading /Users/adrian/Dropbox/Projects/jinnie/genie.jl, in > expression starting on line 16 > > luni, 12 septembrie 2016, 23:01:51 UTC+2, Adrian Salceanu a scris: >> >> I was wondering if anybody can point me towards a tutorial or a large >> code base using parallel computing. Everything that is discussed so far in >> the docs and books is super simple - take a function, run it in parallel, >> the end. >> >> To explain, I'm working on a full stack MVC web framework - so think many >> functions and a few types grouped in maybe 20 modules. Plus more or less 20 >> other external modules. The workflow that I'm after is: >> >> 1. bootstrap - load the necessary components to start up the framework >> (parsing command line args, loading configuration, setting up include >> paths, etc) >> 2. start an instance of HttpServer and listen to a port >> 3. when the server receives a request it invokes a function of the Router >> module which is the entry point into the MVC stack >> 4. once the Router gets the request, it's pushed up the MVC stack and at >> the end a HttpServer.Response instance is returned >> >> That being said, >> a. my strategy is simple: for each request, spawn the function call at #3 >> to a worker >> b. imagine that what's at #4 represents 80% of the app, with a multitude >> of functions being invoked across a multitude of modules (ORM, controller, >> Models, Loggers, Databases, Caching, Sessions, Authentication, etc). >> >> Everything works great single process, but getting the stack to run on >> multiple workers it's a nightmare. I got it to the point where at #3 I can >> invoke a simple function call (something like returning a string or a date) >> and run it on multiple workers. But when I try to invoke the Router and >> snowball the framework, I end up in an avalanche of unknown references. >> >> The codebase is now littered with calls to @everywhere that add a lot of >> noise. But up to this point I still wasn't able to make it work. The errors >> come from deep within the Julia stack, the workers crash saying that a >> module or function can't be found but there's no stack trace to point me >> towards the location so it's really trial end error commenting and >> uncommenting "include" and "using" statements to see what gives, I get >> errors from within external modules, etc. >> >> I guess what I'm trying to say is that my experience with parallel >> computing applied to a large codebase is very frustrating. And I was >> wondering if anybody has done a parallel implementation in a larger >> codebase and has any tips on how to attack it. >> >> >> P.S. >> I might be spoiled by Elixir, but ideally I'd like to be able to spawn >> processes and functions whenever I need, and have the compiler take care of >> making the code available across workers. >> Another thing that would be very useful, also inspired by Elixir, is the >> supervisor design pattern, to look over and restart the workers when they >> crash. >> >
