Yeah, thanks for the clarification about GIL, that was awesome (I read many a textbook that was not as well written).
Made me realise that you can do some calls to the Python's C APY from multiple C threads, but you should do so seldomly as its more expensive in terms of interruptions in the parallelism (given that there is a single lock on the entire API). On May 21, 6:32 am, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > On May 21, 8:14 pm, Magnitus <eric_vallee2...@yahoo.ca> wrote: > > > Now that you mention it, I recall reading in the Python/C API that > > Python wasn't really thread-safe and that Python objects shouldn't be > > accessed from multiple C threads (they recommended using the Python > > threading API which was exposed in the Python/C API instead, but that > > didn't interest me as its not true multi-threading). > > Python is thread safe so long as you abide by the contract of > acquiring the Python GIL. If you are going to ignore that required > contract, then obviously it will break. > > This is the norm for any Python system in as much as you need to > acquire a lock when accessing a shared resource. In the Python case > there so happens to be one single global lock around any C API > accesses where as in a custom C/C++ application which is > multithreaded, you might have more fine grained locking around > specific data structures of parts of the API. > > > My solution to this was make my C++ code C++ only and then glue parts > > that I wanted to expose to Python so that Python calls on the C++ > > compiled code, but not the reverse. > > Which is exactly what many C extension modules do. That is, they move > data between Python and C worlds so that they can then release the GIL > and process the data. That why it can still make use of multi core/ > processor systems. This is why Apache/mod_wsgi isn't constrained by > the Python GIL because everything Apache itself does doesn't use the > Python GIL and can be properly parallelised. > > Graham > > > > > Hence, I probably bypassed a large part of the problematic discussed > > above by not delving deeply into Python's threading API. > > > On May 21, 5:00 am, Yarko Tymciurak <resultsinsoftw...@gmail.com> > > wrote: > > > > On May 21, 3:33 am, Magnitus <eric_vallee2...@yahoo.ca> wrote: > > > > > But if you create "tasks" without doing it at the OS level, doesn't > > > > that means that you won't really be able to take full advantage of > > > > multi-processor hardware (since the OS handles the hardware and if the > > > > OS doesn't know about it, it won't be able to do the required > > > > optimizations with the hardware)? > > > > With the GIL, python itself does not utilize multiple processors, so > > > web2py is processor-bound (the only > > > effect of multi-core is that the o/s itself can "leave" a core to the > > > main python task, e.g. > > > it can grab an alternate core... other than that, you're running on > > > one core regardless - > > > unless you fire multiple instances of python interpreters, in which > > > case you are really only > > > going to communicate thru services anyway.... > > > > See some of the discussion > > > athttp://bugs.python.org/issue7946,http://stackoverflow.com/questions/9...... > > > > ... and so forth... > > > > > Maybe I've done C/C++ for too long and am trying to micro-manage too > > > > much, but a solution to I like to use for the problem of creating/ > > > > tearing down process threads is just to pre-create a limited number of > > > > them (optimised for the number of CPUs you have) and recycle them to > > > > do various tasks as needed. > > > > Well - since you don't have that with python, you run the risk of I/O > > > blocking .... which is why really lightweight > > > tasklets are so desireable (CCP Games > > > runshttp://en.wikipedia.org/wiki/Eve_Online > > > with many tens of thousands of simultaneous users, if I recall > > > correctly, and maintain stackless for this purpose). > > > > > Of course, that works best when you give your threads/processes longer > > > > tasks to perform in parallel (else, the extra cost of managing it will > > > > probably outweight the benefits of running it in parallel). > > > > There is much to cover in this - and I suppose reason to be happy that > > > python traditionally hasn't run multi-core. > > > See, for example, the discussions > > > at:http://stackoverflow.com/questions/203912/does-python-support-multipr... > > > > andhttp://docs.python.org/library/multiprocessing.html > > > > Lots to read! ;-) > > > > - Yarko > > > > > On May 20, 2:12 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > > > > wrote: > > > > > > On May 19, 6:18 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com> > > > > > wrote: > > > > > > > On May 19, 5:41 pm, amoygard <amoyg...@gmail.com> wrote: > > > > > > .... > > > > > > > So - in general, you do not start subprocesses - with the exception > > > > > > of > > > > > > cron. Seehttp://www.web2py.com/book/default/section/4/17 > > > > > > I might better have said you do not _want_ to be starting subprocesses > > > > > - besides the cost (compute time, memory, etc.), if you generally did > > > > > this. This (the inneficiency of spawning subrocesses) is why > > > > > stackless was created - and (among other things) used in a a very > > > > > busy online game. A lot of thought went into avoiding the costs of > > > > > spawning subprocesses. > > > > > > If you haven't heard of it, stackless is an implementation of python > > > > > that does not use the traditional "C" stack for local variables, > > > > > etc. Among other things, it has added "tasklets" to the language, so > > > > > you can create and schedule tasks - without the overhead of doing so > > > > > in your operating system. There is a lot of discussion of benefit, > > > > > efficiency. Although there might be some discussion questioning the > > > > > approach, other alternative approaches, one thing is clear: the > > > > > motivation to stay away from creating threads / subprocesses, and the > > > > > costs involved. it might be interesting to read up on it. > > > > > > - Yarko > > > > > > > - Yarko- Hide quoted text - > > > > - Show quoted text -- Hide quoted text - > > - Show quoted text -