Yeah, thanks for the clarification about GIL, that was awesome (I read
many a textbook that was not as well written).

Made me realise that you can do some calls to the Python's C APY from
multiple C threads, but you should do so seldomly as its more
expensive in terms of interruptions in the parallelism (given that
there is a single lock on the entire API).

On May 21, 6:32 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> On May 21, 8:14 pm, Magnitus <eric_vallee2...@yahoo.ca> wrote:
>
> > Now that you mention it, I recall reading in the Python/C API that
> > Python wasn't really thread-safe and that Python objects shouldn't be
> > accessed from multiple C threads (they recommended using the Python
> > threading API which was exposed in the Python/C API instead, but that
> > didn't interest me as its not true multi-threading).
>
> Python is thread safe so long as you abide by the contract of
> acquiring the Python GIL. If you are going to ignore that required
> contract, then obviously it will break.
>
> This is the norm for any Python system in as much as you need to
> acquire a lock when accessing a shared resource. In the Python case
> there so happens to be one single global lock around any C API
> accesses where as in a custom C/C++ application which is
> multithreaded, you might have more fine grained locking around
> specific data structures of parts of the API.
>
> > My solution to this was make my C++ code C++ only and then glue parts
> > that I wanted to expose to Python so that Python calls on the C++
> > compiled code, but not the reverse.
>
> Which is exactly what many C extension modules do. That is, they move
> data between Python and C worlds so that they can then release the GIL
> and process the data. That why it can still make use of multi core/
> processor systems. This is why Apache/mod_wsgi isn't constrained by
> the Python GIL because everything Apache itself does doesn't use the
> Python GIL and can be properly parallelised.
>
> Graham
>
>
>
> > Hence, I probably bypassed a large part of the problematic discussed
> > above by not delving deeply into Python's threading API.
>
> > On May 21, 5:00 am, Yarko Tymciurak <resultsinsoftw...@gmail.com>
> > wrote:
>
> > > On May 21, 3:33 am, Magnitus <eric_vallee2...@yahoo.ca> wrote:
>
> > > > But if you create "tasks" without doing it at the OS level, doesn't
> > > > that means that you won't really be able to take full advantage of
> > > > multi-processor hardware (since the OS handles the hardware and if the
> > > > OS doesn't know about it, it won't be able to do the required
> > > > optimizations with the hardware)?
>
> > > With the GIL, python itself does not utilize multiple processors, so
> > > web2py is processor-bound (the only
> > > effect of multi-core is that the o/s itself can "leave" a core to the
> > > main python task, e.g.
> > > it can grab an alternate core... other than that, you're running on
> > > one core regardless -
> > > unless you fire multiple instances of python interpreters, in which
> > > case you are really only
> > > going to communicate thru services anyway....
>
> > > See some of the discussion 
> > > athttp://bugs.python.org/issue7946,http://stackoverflow.com/questions/9......
>
> > > ... and so forth...
>
> > > > Maybe I've done C/C++ for too long and am trying to micro-manage too
> > > > much, but a solution to I like to use for the problem of creating/
> > > > tearing down process threads is just to pre-create a limited number of
> > > > them (optimised for the number of CPUs you have) and recycle them to
> > > > do various tasks as needed.
>
> > > Well - since you don't have that with python, you run the risk of I/O
> > > blocking .... which is why really lightweight
> > > tasklets are so desireable (CCP Games 
> > > runshttp://en.wikipedia.org/wiki/Eve_Online
> > > with many tens of thousands of simultaneous users, if I recall
> > > correctly, and maintain stackless for this purpose).
>
> > > > Of course, that works best when you give your threads/processes longer
> > > > tasks to perform in parallel (else, the extra cost of managing it will
> > > > probably outweight the benefits of running it in parallel).
>
> > > There is much to cover in this - and I suppose reason to be happy that
> > > python traditionally hasn't run multi-core.
> > > See, for example, the discussions 
> > > at:http://stackoverflow.com/questions/203912/does-python-support-multipr...
>
> > > andhttp://docs.python.org/library/multiprocessing.html
>
> > > Lots to read! ;-)
>
> > > - Yarko
>
> > > > On May 20, 2:12 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com>
> > > > wrote:
>
> > > > > On May 19, 6:18 pm, Yarko Tymciurak <resultsinsoftw...@gmail.com>
> > > > > wrote:
>
> > > > > > On May 19, 5:41 pm, amoygard <amoyg...@gmail.com> wrote:
>
> > > > > ....
>
> > > > > > So - in general, you do not start subprocesses - with the exception 
> > > > > > of
> > > > > > cron.   Seehttp://www.web2py.com/book/default/section/4/17
>
> > > > > I might better have said you do not _want_ to be starting subprocesses
> > > > > - besides the cost (compute time, memory, etc.), if you generally did
> > > > > this.   This (the inneficiency of spawning subrocesses) is why
> > > > > stackless  was created - and (among other things) used in a a very
> > > > > busy online game.  A lot of thought went into avoiding the costs of
> > > > > spawning subprocesses.
>
> > > > > If you haven't heard of it, stackless is an implementation of python
> > > > > that does not use the traditional "C" stack for local variables,
> > > > > etc.   Among other things, it has added "tasklets" to the language, so
> > > > > you can create and schedule tasks - without the overhead of doing so
> > > > > in your operating system.   There is a lot of discussion of benefit,
> > > > > efficiency.   Although there might be some discussion questioning the
> > > > > approach, other alternative approaches, one thing is clear:  the
> > > > > motivation to stay away from creating threads / subprocesses, and the
> > > > > costs involved.  it might be interesting to read up on it.
>
> > > > > - Yarko
>
> > > > > > - Yarko- Hide quoted text -
>
> > > - Show quoted text -- Hide quoted text -
>
> - Show quoted text -

Reply via email to