You are describing what I see as a need to run processes asynchronously. We use the Torque queue system for this. It takes care of scheduling and running jobs on a cluster.
On Thu, Apr 17, 2014 at 6:52 AM, Christoph Groth <christ...@grothesque.org>wrote: > Hi Julian, > > > Generally, org-babel is suited for long running computations. Its > > caching can help you avoid rerunning code chunks. And long runtime does > > not conflict with the idea of reproducible research, it just may be not > > very comfortable for the user. > > I agree of course that it’s generally a good idea to structure the > analysis in small logical steps and to save intermediate results. Let > me narrow down my question then: does org-babel support working with > tasks that take several minutes to execute? By this I mean (most > important first): > > - Not freezing the editor during the execution of a task > You code-block should exit if a job is submitted, and record the job-id so that you can check it later. Alternatively, you could do this in ipython notebooks if you have python functions that return jobids. > - Being able to execute multiple independent tasks in parallel > No problem. this is what queue systems were designed for. > - Being able to interrupt a running task > you do this with the queue commands, e.g. qdel jobid > - Being able to inspect the incomplete output of a running task > This is just checking the output files in the running jobs directories. > > I’d love to hear about any frameworks or workflows that fulfill these > requirements. > > Cheers > > Christoph > > > PS. > > I’m using ipython notebooks but I’m not happy with them because of the > freezing problem and the complete lack of isolation of tasks within a > single notebook (they live in a common mutable namespace). I think a > useful framework must be “functional” at the highest level for caching > and dependencies to be useful. > > >
