Shane Hathaway wrote: > robert wrote: > > I'd like to use multiple CPU cores for selected time consuming Python > > computations (incl. numpy/scipy) in a frictionless manner. > > > > Interprocess communication is tedious and out of question, so I > > thought about simply using a more Python interpreter instances > > (Py_NewInterpreter) with extra GIL in the same process. I expect to > > be able to directly push around Python Object-Trees between the 2 (or > > more) interpreters by doing some careful locking. > > > > Any hope to come through? If possible, what are the main dangers? Is > > there an example / guideline around for that task? - using ctypes or > > so. > > > > Or is there even a ready made Python module which makes it easy to > > setup and deal with extra Interpreter instances? If not, would it be > > an idea to create such thing in the Python std libs to make Python > > multi-processor-ready. I guess Python will always have a GIL - > > otherwise it would loose lots of comfort in threaded programming > > I'd like to mention mod_python, which creates multiple interpreters > inside Apache. It does this transparently and works well. There is no > apparent lock contention between the interpreters, because no Python > objects are shared.
In relation to mod_python, your statement isn't strictly true. One can create scenarios in mod_python where a Python object created in the context of one interpreter is then used in another interpreter at a later point. Generally this isn't an issue because it is part of a recursive sequence of processing and thus while the second interpreter is doing stuff with the object the first is stalled waiting for the second to return. It wouldn't be totally inconceivable though for a user to create threads which may operate on the object at the same time in the first interpreter as the object is being used in the second interpreter. The situation gets even more complicated when you start to use third party modules which may be used from multiple interpreters and a multithreaded Apache MPM is used and where the third party module performs callbacks into interpreters with some common object. Thus, mod_python may not be able to benefit from each Python interpreter instance having its own GIL if that was your point in referencing mod_python. Graham -- http://mail.python.org/mailman/listinfo/python-list
