[Python-Dev] Multiple interpreters not compatible with current thread module
The current threadmodule.c does not seem to correctly support multiple
(sub) interpreters.
This became apparent when using jep - (a Java/Python bridge) and also
seems to cause problems with mod_python.
The incompatibility began in Python version 2.3.5 and has been traced to changes
to the 2.4 threadmodule.c that were backported to the 2.3 delivery.
A bug report was raised on 2005-03-15
(http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470)
which covers the problem in more detail.
I've just submitted a patch (I hope it's correctly formatted) for
threadmodule.c
(http://sourceforge.net/tracker/index.php?func=detail&aid=1203393&group_id=5470&atid=305470)
adapted from the pre-2.3.5 threadmodule.c (replacing the
PyGILState_XXX calls with those from the earlier thread module).
The patch works correctly but it will probably re-introduce the
problem that the change for threadmodule.c version 2.59 fixed.("Fix
for [ 1010677 ] thread Module Breaks PyGILState_Ensure(),and a test
case.When booting a new thread, use the PyGILState API to manage the
GIL.").
The documentation (http://docs.python.org/api/threads.html) states
"Note that the PyGILState_*() functions assume there is only one
global interpreter (created automatically by Py_Initialize()). Python
still supports the creation of additional interpreters (using
Py_NewInterpreter()), but mixing multiple interpreters and the
PyGILState_*() API is unsupported. ", so it looks like that using the
PyGilState_XXX functions in the core threadmodule.c means the
Py_NewInterpreter() call (i.e. multiple interpreters) is no longer
supported when threads are involved.
This problem is preventing us upgrading to Python 2.4 so we'd
obviously like to see a resolution the next Python release if that
were possible...
Cheers,
Max
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Vestigial code in threadmodule?
If you're digging into the threadmodule.c could you take a look at
bug #1163563
(http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470)
I've posted a patch
(http://sourceforge.net/tracker/index.php?
func=detail&aid=1203393&group_id=5470&atid=305470)
Regards,
Max
On 6/2/05, A.M. Kuchling <[EMAIL PROTECTED]> wrote:
> Looking at bug #1209880, the following function from threadmodule.c is
> referenced. I think the args==NULL case, which can return None
> instead of a Boolean value, can never be reached because
> PyArg_ParseTuple() will fail if args==NULL.
>
> Before ripping the args==NULL code out, I wanted to be sure my
> analysis is correct; is there some subtlety here I'm missing that
> makes args==NULL possible?
>
> --amk
>
> static PyObject *
> lock_PyThread_acquire_lock(lockobject *self, PyObject *args)
> {
> int i = 1;
>
> if (!PyArg_ParseTuple(args, "|i:acquire", &i))
> return NULL;
>
> Py_BEGIN_ALLOW_THREADS
> i = PyThread_acquire_lock(self->lock_lock, i);
> Py_END_ALLOW_THREADS
>
> if (args == NULL) {
> Py_INCREF(Py_None);
> return Py_None;
> }
> else
> return PyBool_FromLong((long)i);
> }
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anothermax%40gmail.com
>
--
flickr: http://www.flickr.com/photos/anothermax/sets
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IMPORTANT: release24-maint branch is FROZEN from 2005-09-21 00:00 UTC for 2.4.2
Can the fix for [ 1163563 ] Sub threads execute in restricted mode (http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470 ) go in before 2.4.2 final? This is a real show stopper for us - we can't move to 2.4 without it. Cheers, Max On 9/22/05, Michael Hudson <[EMAIL PROTECTED]> wrote: Reinhold Birkenfeld <[EMAIL PROTECTED] > writes:> Anthony Baxter wrote:>> Starting in about 11 hours time, the release24-maint branch is FROZEN>> for the 2.4.2c1 release. The freeze will last for around a day, and>> then we're in a state of mostly-frozen for another week, until 2.4.2>> (final). During that week, please don't check things into the branch>> unless you check with me first. Let's make this a nice painless>> release. I'll send another message once 2.4.2 is done.>> Do you think the patch at>> https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1298449&group_id=5470 >> could go in before 2.4.2 final? It's a rather major issue> within cStringIO.writelines().Oh, I was looking at that one too :) I was going to check it in afterbuilding and testing finished... Cheers,mwh--The gripping hand is really that there are morons everywhere, it'sjust that the Americon morons are funnier than average. -- Pim van Riezen, alt.sysadmin.recovery ___Python-Dev mailing [email protected]://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/anothermax%40gmail.com -- flickr: http://www.flickr.com/photos/anothermax/sets ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IMPORTANT: release24-maint branch is FROZEN from 2005-09-21 00:00 UTC for 2.4.2
I don't think it's too risky. Michael Hudson did the work and it's been checked into the python CVS for a while. (Python/pystate.c Rev 2.42 and Modules/threadmodule.c Rev 2.64 are required files). We're stuck on Python 2.3.3 because of this bug...(2.4 thread code with this bug was backported to 2.3.5). Max On 9/22/05, Anthony Baxter <[EMAIL PROTECTED]> wrote: On Thursday 22 September 2005 23:36, Jeremy Maxfield wrote:> Can the fix for [ 1163563 ] Sub threads execute in restricted mode > (> http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&gr>oup_id=5470&atid=105470 ) > go in before 2.4.2 final?> This is a real show stopper for us - we can't move to 2.4 without> it. Cheers,> MaxI'm not sure - mwh, how risky a patch is this?Anthony--Anthony Baxter < [EMAIL PROTECTED]>It's never too late to have a happy childhood.-- flickr: http://www.flickr.com/photos/anothermax/sets ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IMPORTANT: release24-maint branch is FROZEN from 2005-09-21 00:00 UTC for 2.4.2
Sorry I think you're 'much mistaken'... The revision of PyState.c in the rc242c1 looks like 2.38.22 (should be 2.42) and threadmodule.c looks like 2.59 (should be 2.64) Cheers, Max On 9/22/05, Michael Hudson <[EMAIL PROTECTED]> wrote: Anthony Baxter <[EMAIL PROTECTED]> writes: > On Thursday 22 September 2005 23:36, Jeremy Maxfield wrote:>> Can the fix for [ 1163563 ] Sub threads execute in restricted mode>> (>> http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&gr>>oup_id=5470&atid=105470 )>> go in before 2.4.2 final?>> This is a real show stopper for us - we can't move to 2.4 without>> it. Cheers,>> Max>> I'm not sure - mwh, how risky a patch is this?Well, unless I'm much mistaken it's in 2.4.2c1Cheers,mwh--I think if we have the choice, I'd rather we didn't explicitly put flaws in the reST syntax for the sole purpose of not insulting thealmighty.-- /will on the doc-sig___Python-Dev mailing list [email protected]://mail.python.org/mailman/listinfo/python-devUnsubscribe: http://mail.python.org/mailman/options/python-dev/anothermax%40gmail.com-- flickr: http://www.flickr.com/photos/anothermax/sets ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IMPORTANT: release24-maint branch is FROZEN from 2005-09-21 00:00 UTC for 2.4.2
That's great - thanks alot. Cheers, Max On 9/23/05, Michael Hudson <[EMAIL PROTECTED]> wrote: Michael Hudson <[EMAIL PROTECTED]> writes:> Jeremy Maxfield < [EMAIL PROTECTED]> writes:>>> Sorry I think you're 'much mistaken'...>>>> The revision of PyState.c in the rc242c1 looks like 2.38.22 (should be >> 2.42)>> and threadmodule.c looks like 2.59 (should be 2.64)>> Looks like you're right but SF CVS is being *astonishingly* slow right> now. The fixes will be in 2.4.2.OK, I've finally managed to get these changes checked in. Cheers,mwh-- . <- the pointyour article -> . |- a long way | -- Christophe Rhodes, ucam.chat___Python-Dev mailing [email protected]://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/anothermax%40gmail.com -- flickr: http://www.flickr.com/photos/anothermax/sets ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency
Support for multiple interpreters already exists from the C API
(mod_python, Java Embedded Python a few other add-ons use them)
But:
- it's not possible to create new interpreter instances from within Python.
- there's no mechanism for passing information between interpreters.
- interaction with extension modules instances may be a problem.
Apart from these points they actually seem to work pretty well and it might be, as
you suggest, a "Practical" approach.
Implementing a 'subinterp' module could be interesting...
Max
On 9/30/05, Paul Moore <[EMAIL PROTECTED]> wrote:
On 9/30/05, Jim Jewett <[EMAIL PROTECTED]> wrote:> Bruce Eckel wrote:
>> > 3) Tasks are cheap enough that I can make> > thousands of them, ...>> > 4) Tasks are "self-guarding," so they prevent> > other tasks from interfering with them. The
> > only way tasks can communicate with each> > other is through some kind of formal> > mechanism (something queue-ish,> > I'd imagine).>> I think these two are the hardest to reconcile.
Agreed. I think the biggest problem is that (4) is too strong. At theOS level, certain issues (such as the current directory, or the stdiostreams) are per-process, so that taking (4) at face value requires
multiprocessing, which violates (3)...If you constrain (4), you can get something much more effective. Forexample, if you accept that certain things are volatile (start withOS-specified per-process stuff, plus the Python builtins, maybe) then
it's much easier to produce solutions.I don't think that shared state is that big an issue per se - afterall, we're all used to the idea that the contents of a file mightchange "behind our backs". The real issue is not having a *clearly
defined* shared state.The big problem with threads is that the shared state is *everything*.There's no partitioning at all. Every thread-based abstraction isbased around threads voluntarily restricting themselves to a limited
set of "safe" communication operations, and otherwise scrupulouslyavoiding each other's space ("Don't sit there, that's auntie Mary'schair"...)If we had an abstraction (multiple interpreters, anyone?) which still
had a shared state, but a much smaller one, which was defined clearly,such that "unsafe" operations were easy to identify, then day-to-dayconcurrent programming would be a lot easier. Yes, it's still possible
to break things, but people can do that already by hacking about withbuiltins, or abusing the introspection API, and it's not made Pythonunusable, because they generally don't...This seems to me to be a perfect case for a "Practicality beats
Purity" approach.Paul.___Python-Dev mailing [email protected]
http://mail.python.org/mailman/listinfo/python-devUnsubscribe: http://mail.python.org/mailman/options/python-dev/anothermax%40gmail.com
-- flickr: http://www.flickr.com/photos/anothermax/sets
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sharing between multiple interpreters and restricted mode
Hi Gabriel, Apart from the problem of sharing code between interpreters (sounds pretty dangerous to me) there's a bug in 2.4.1 (and 2.3.5 for that matter) that causes problems with multiple interpreters in certain siutations (esp. threads) which results in this 'restricted mode' message - I encountered it when trying to do some work with JEP (Java Embedded Python). Python 2.4.2 fixed this problem for me. I would suggest giving 2.4.2 a try too. Good luck, Max On 1/4/06, Gabriel Becedillas <[EMAIL PROTECTED]> wrote: > Hi, > At the company I work for we've embedded Python 2.4.1 in a C++ > application. We execute multiple scripts concurrenlty, each one in its > own interpreter (created using Py_NewInterpreter()). > We are sharing a certain instance between interpreters because its to > expensive to instantiate that class every time an interpreter is > created. The class is instantiated in the main interpreter (that is > always alive) and every time a new interpreter is created, that > instance is added to the interpreter's __builtins__ dict. > The problem I'm having is that some methods of that class (when > executed in an interpreter different from the one it was created in) > complain about being in restricted execution mode. > Assuming that there are no issues with this instance lifetime (coz it > will always be referenced by the main interpreter), is there a safe way > to do some sharing between interpreters ?. > Thanks. > > -- > > Gabriel Becedillas > Developer > CORE SECURITY TECHNOLOGIES > > Florida 141 - 2º cuerpo - 7º piso > C1005AAC Buenos Aires - Argentina > Tel/Fax: (54 11) 5032-CORE (2673) > http://www.corest.com > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anothermax%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
