Now that you mention it, I recall reading in the Python/C API that Python wasn't really thread-safe and that Python objects shouldn't be accessed from multiple C threads (they recommended using the Python threading API which was exposed in the Python/C API instead, but that didn't interest me as its not true multi-threading).
My solution to this was make my C++ code C++ only and then glue parts that I wanted to expose to Python so that Python calls on the C++ compiled code, but not the reverse. Hence, I probably bypassed a large part of the problematic discussed above by not delving deeply into Python's threading API. On May 21, 5:00 am, Yarko Tymciurak <resultsinsoftw...@gmail.com> wrote: > On May 21, 3:33 am, Magnitus <eric_vallee2...@yahoo.ca> wrote: > > > But if you create "tasks" without doing it at the OS level, doesn't > > that means that you won't really be able to take full advantage of > > multi-processor hardware (since the OS handles the hardware and if the > > OS doesn't know about it, it won't be able to do the required > > optimizations with the hardware)? > > With the GIL, python itself does not utilize multiple processors, so > web2py is processor-bound (the only > effect of multi-core is that the o/s itself can "leave" a core to the > main python task, e.g. > it can grab an alternate core... other than that, you're running on > one core regardless - > unless you fire multiple instances of python interpreters, in which > case you are really only > going to communicate thru services anyway.... > > See some of the discussion > athttp://bugs.python.org/issue7946,http://stackoverflow.com/questions/990102/python-global-interpreter-l... > > ... and so forth... > > > > > Maybe I've done C/C++ for too long and am trying to micro-manage too > > much, but a solution to I like to use for the problem of creating/ > > tearing down process threads is just to pre-create a limited number of > > them (optimised for the number of CPUs you have) and recycle them to > > do various tasks as needed. > > Well - since you don't have that with python, you run the risk of I/O > blocking .... which is why really lightweight > tasklets are so desireable (CCP Games > runshttp://en.wikipedia.org/wiki/Eve_Online > with many tens of thousands of simultaneous users, if I recall > correctly, and maintain stackless for this purpose). > > > > > Of course, that works best when you give your threads/processes longer > > tasks to perform in parallel (else, the extra cost of managing it will > > probably outweight the benefits of running it in parallel). > > There is much to cover in this - and I suppose reason to be happy that > python traditionally hasn't run multi-core. > See, for example, the discussions > at:http://stackoverflow.com/questions/203912/does-python-support-multipr... > > andhttp://docs.python.org/library/multiprocessing.html > > Lots to read! ;-) > > - Yarko > > > > > > > On May 20, 2:12 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > > wrote: > > > > On May 19, 6:18 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > > > wrote: > > > > > On May 19, 5:41 pm, amoygard <amoyg...@gmail.com> wrote: > > > > .... > > > > > So - in general, you do not start subprocesses - with the exception of > > > > cron. Seehttp://www.web2py.com/book/default/section/4/17 > > > > I might better have said you do not _want_ to be starting subprocesses > > > - besides the cost (compute time, memory, etc.), if you generally did > > > this. This (the inneficiency of spawning subrocesses) is why > > > stackless was created - and (among other things) used in a a very > > > busy online game. A lot of thought went into avoiding the costs of > > > spawning subprocesses. > > > > If you haven't heard of it, stackless is an implementation of python > > > that does not use the traditional "C" stack for local variables, > > > etc. Among other things, it has added "tasklets" to the language, so > > > you can create and schedule tasks - without the overhead of doing so > > > in your operating system. There is a lot of discussion of benefit, > > > efficiency. Although there might be some discussion questioning the > > > approach, other alternative approaches, one thing is clear: the > > > motivation to stay away from creating threads / subprocesses, and the > > > costs involved. it might be interesting to read up on it. > > > > - Yarko > > > > > - Yarko- Hide quoted text - > > - Show quoted text -
