On May 21, 3:33 am, Magnitus <eric_vallee2...@yahoo.ca> wrote: > But if you create "tasks" without doing it at the OS level, doesn't > that means that you won't really be able to take full advantage of > multi-processor hardware (since the OS handles the hardware and if the > OS doesn't know about it, it won't be able to do the required > optimizations with the hardware)?
With the GIL, python itself does not utilize multiple processors, so web2py is processor-bound (the only effect of multi-core is that the o/s itself can "leave" a core to the main python task, e.g. it can grab an alternate core... other than that, you're running on one core regardless - unless you fire multiple instances of python interpreters, in which case you are really only going to communicate thru services anyway.... See some of the discussion at http://bugs.python.org/issue7946, http://stackoverflow.com/questions/990102/python-global-interpreter-lock-gil-workaround-on-multi-core-systems-using-tasks ... and so forth... > > Maybe I've done C/C++ for too long and am trying to micro-manage too > much, but a solution to I like to use for the problem of creating/ > tearing down process threads is just to pre-create a limited number of > them (optimised for the number of CPUs you have) and recycle them to > do various tasks as needed. Well - since you don't have that with python, you run the risk of I/O blocking .... which is why really lightweight tasklets are so desireable (CCP Games runs http://en.wikipedia.org/wiki/Eve_Online with many tens of thousands of simultaneous users, if I recall correctly, and maintain stackless for this purpose). > > Of course, that works best when you give your threads/processes longer > tasks to perform in parallel (else, the extra cost of managing it will > probably outweight the benefits of running it in parallel). There is much to cover in this - and I suppose reason to be happy that python traditionally hasn't run multi-core. See, for example, the discussions at: http://stackoverflow.com/questions/203912/does-python-support-multiprocessor-multicore-programming and http://docs.python.org/library/multiprocessing.html Lots to read! ;-) - Yarko > > On May 20, 2:12 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > wrote: > > > On May 19, 6:18 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > > wrote: > > > > On May 19, 5:41 pm, amoygard <amoyg...@gmail.com> wrote: > > > .... > > > > So - in general, you do not start subprocesses - with the exception of > > > cron. Seehttp://www.web2py.com/book/default/section/4/17 > > > I might better have said you do not _want_ to be starting subprocesses > > - besides the cost (compute time, memory, etc.), if you generally did > > this. This (the inneficiency of spawning subrocesses) is why > > stackless was created - and (among other things) used in a a very > > busy online game. A lot of thought went into avoiding the costs of > > spawning subprocesses. > > > If you haven't heard of it, stackless is an implementation of python > > that does not use the traditional "C" stack for local variables, > > etc. Among other things, it has added "tasklets" to the language, so > > you can create and schedule tasks - without the overhead of doing so > > in your operating system. There is a lot of discussion of benefit, > > efficiency. Although there might be some discussion questioning the > > approach, other alternative approaches, one thing is clear: the > > motivation to stay away from creating threads / subprocesses, and the > > costs involved. it might be interesting to read up on it. > > > - Yarko > > > > - Yarko
