But if you create "tasks" without doing it at the OS level, doesn't that means that you won't really be able to take full advantage of multi-processor hardware (since the OS handles the hardware and if the OS doesn't know about it, it won't be able to do the required optimizations with the hardware)?
Maybe I've done C/C++ for too long and am trying to micro-manage too much, but a solution to I like to use for the problem of creating/ tearing down process threads is just to pre-create a limited number of them (optimised for the number of CPUs you have) and recycle them to do various tasks as needed. Of course, that works best when you give your threads/processes longer tasks to perform in parallel (else, the extra cost of managing it will probably outweight the benefits of running it in parallel). On May 20, 2:12 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> wrote: > On May 19, 6:18 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > wrote: > > > On May 19, 5:41 pm, amoygard <amoyg...@gmail.com> wrote: > > .... > > > So - in general, you do not start subprocesses - with the exception of > > cron. Seehttp://www.web2py.com/book/default/section/4/17 > > I might better have said you do not _want_ to be starting subprocesses > - besides the cost (compute time, memory, etc.), if you generally did > this. This (the inneficiency of spawning subrocesses) is why > stackless was created - and (among other things) used in a a very > busy online game. A lot of thought went into avoiding the costs of > spawning subprocesses. > > If you haven't heard of it, stackless is an implementation of python > that does not use the traditional "C" stack for local variables, > etc. Among other things, it has added "tasklets" to the language, so > you can create and schedule tasks - without the overhead of doing so > in your operating system. There is a lot of discussion of benefit, > efficiency. Although there might be some discussion questioning the > approach, other alternative approaches, one thing is clear: the > motivation to stay away from creating threads / subprocesses, and the > costs involved. it might be interesting to read up on it. > > - Yarko > > > > > - Yarko
