Bugs item #1163563, was opened at 2005-03-15 08:56 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1163563&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Threads Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 8 Submitted By: anothermax (yetanothermax) >Assigned to: Michael Hudson (mwh) Summary: Sub threads execute in restricted mode Initial Comment: I'm using the JEP product which allows integration of Java with Python (see http://jepp.sourceforge.net) via starting a Python interpreter in the same process as the JVM. This integrates with python via the C interface, allowing the user to 'eval' python code (amongst other features). It seems that since Python 2.3.5 any threads (using the threading module) created via code that has been evaluated through the jep.eval() interface (i.e.Python C interface )are executed in restricted mode and cannot, for example, create files. Code that is just evaled (i.e not in a subthread) thread has no restrictions. The error reported is: IOError: file() constructor not accessible in restricted mode (see http://sourceforge.net/forum/forum.php? thread_id=1247793&forum_id=376782) - I've given a JEP specific example here. There seems to be a similar problem with mod_python (see http://www.modpython.org/pipermail/mod_python/2005- January/017129.html) Reading through the release notes for Python 2.3.5 I see: Bug #754449: threading.Thread will no longer mask exceptions raised during interpreter shutdown with another exception caused by attempting to output the initial exception. This fix also includes a backport of rev. 1.41 from HEAD. This might be the problem as it seems to involve the porting of 2.4 threading code back to the 2.3 tree. I've attached a Java file which shows the problem when using JEP. The error output is: Exception in thread Thread-1: Traceback (most recent call last): File "C:\Python24\Lib\threading.py", line 442, in __bootstrap File "<string>", line 5, in run IOError: file() constructor not accessible in restricted mode 2.4.1c1 (#63, Mar 10 2005, 10:36:41) [MSC v.1310 32 bit (Intel)] Creating file from main thread... Done Creating file from sub thread... Done ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2005-06-20 17:54 Message: Logged In: YES user_id=6656 Fixed, in Misc/NEWS revision 1.1301 Modules/threadmodule.c revision 2.64 Python/pystate.c revision 2.42 (I made a tiny tweak in addition to the latest patch attached to this report). yetanothermax: Thanks for the report, and the testing of patches! ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-06-17 15:37 Message: Logged In: YES user_id=1218811 The maybe-even-better-1010677-fix also seems to work ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-17 13:10 Message: Logged In: YES user_id=6656 That's good to hear. Tim: could you glance over the attached? If you can't be assed to read all the various reports, I've attempted to summarise the problem(s) here: http://mail.python.org/pipermail/python-dev/2005-June/054258.html If you don't have time, please at least say so... yetanothermax: thanks for the confirmation. Could you also try this cleaner version of the patch? Wrt your patch, it took me too long to work out that it was simply unapplying the earlier changes -- and you don't need to supply a patch for that, even creaky old CVS makes *that* easy :) ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-06-17 11:35 Message: Logged In: YES user_id=1218811 Good news - I've made a couple of tests and the patch seems to fix the problem. With regards to the earlier patch: - apologies for not running the test suite - I feel suitably chastised :) & I'll be sure to run it in the future. - I did note that it would probably re-raise #1010677 (see 2005-04-20 13:51 comment) but guessed it was more important to contribute the patch so that someone who has better thread implementation knowledge could look into it... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-16 13:39 Message: Logged In: YES user_id=6656 So here's my take on a fix to 1010677, relative to CVS HEAD. It's something of a hack, and I certainly wouldn't condone applying it to the trunk as is but I would be interested in hearing if it fixes your problems. (after applying your patch the test suite failed to complete, btw -- please try this yourself next time :) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-16 13:09 Message: Logged In: YES user_id=6656 (Grr, SF hates me today) Why, are you under the impression that that could make a difference here? (hint: it can't). The problem with Py_NewInterpreter was an over-zealous check in PythreadState_Swap(), which I've tweaked. A red herring. On to the real problem... I see the problem, and can now reproduce it with your test code, but isn't your patch just a reversion of the patch that was attached to bug #1010677? Why won't applying it bring that patch back? ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-06-16 12:32 Message: Logged In: YES user_id=1218811 Have you tried the threadtest.c with a debug build after applying the patch to threadmodule.c? The patch is here http://sourceforge.net/tracker/index.php? func=detail&aid=1203393&group_id=5470&atid=305470 Cheers, Max ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-16 10:58 Message: Logged In: YES user_id=6656 Hmm. threadtest.c just crashed for me with a debug build: Fatal Python error: Invalid thread state for this thread Abort trap This is in the call to Py_NewInterpreter() ... (eek). ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-05-17 09:49 Message: Logged In: YES user_id=1218811 I've created a patch for threadmodule.c http://sourceforge.net/tracker/index.php? func=detail&aid=1203393&group_id=5470&atid=305470 ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-05-12 09:48 Message: Logged In: YES user_id=1218811 It looks like with Python versions > 2.3.4 threads are off limits for multiple interpreters. If that's your definition of screwed then I guess the answer's yes. I'm surprised more mod_python users haven't been bitten by this. Shall I raise theproblem on the list (I've already done so on comp.lang.python a while ago) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-05-12 08:43 Message: Logged In: YES user_id=6656 I saw the subject and thought "Hmm, should raise the priority of this so it gets at least thought about hard by the next release", then saw that you did that... > How do we get this solved for the next Python release? I don't know. It's possible it should be raised on python-dev (it seems fairly quiet just now, so it might be a good time :). Basically, it seems that the whole multiple interpreters facility is screwed, correct? ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-05-12 08:35 Message: Logged In: YES user_id=1218811 This problem prevents us from upgrading to Python 2.4. I've attached a another 'fixed' threadmodule.c which is based upon the lastest CVS release.(Only change from the last threadmodule.c attachment is the removal of a '\n' from an error message). The threadtest.c program demonstrates the problem (seems to be the use of Py_NewInterpreter() with the PyGIL_StateXXX functions in the current threadmodule.c). How do we get this solved for the next Python release? ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-04-21 13:05 Message: Logged In: YES user_id=1218811 I've attached a file (threadtest.c) that is adapted from the the way JEP uses Py_NewInterpreter. This runs correctly under 2.3.4 but fails under 2.3.5 and later Pythons. ---------------------------------------------------------------------- Comment By: mrjohnson (mrjohnson0) Date: 2005-04-21 06:51 Message: Logged In: YES user_id=1044885 In interactive mode, Jep uses Py_NewInterpreter, Py_CompileString, and PyRun_String for its eval(). This file handles java -> python: http://204.156.146.230/cgi-bin/viewcvs.cgi/jep/pyembed.c?rev=1.29&content-type=text/vnd.viewcvs-markup You'd be looking for pyembed_eval(). ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2005-04-21 04:11 Message: Logged In: YES user_id=14198 Can anyone tell me specifically what Python C API function is involved here? ie, the "entry point" that these external threads use to call into Python. ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-04-20 12:51 Message: Logged In: YES user_id=1218811 I didn't see jbelmote's comment, perhaps my updated threadmodule.c will reopen the #1010677 bug. ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-04-20 12:39 Message: Logged In: YES user_id=1218811 Here's the updated threadmodule.c for 2.4.1 ---------------------------------------------------------------------- Comment By: anothermax (yetanothermax) Date: 2005-04-20 12:36 Message: Logged In: YES user_id=1218811 I've finally got access to a C compiler and it looks like the 'pure' Python changes for #754449 are in the clear (apologies to bcannon). The problem actually seems to relate to changes to threadmodule.c of the Python Modules directory. In versions greater than 2.3.4 it uses the PyGILState_XXX constructs t_bootstrap() function. These do not seem to be compatible with running sub interpreters as JEP (and probably mod_python) do. The python documentation (http://docs.python.org/api/threads.html) says "Note that the PyGILState_*() functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python still supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_*() API is unsupported. " I've compiled a new versions of the python2X.dll for both 2.3.5 and 2.4.1 changing the use of these PyGIL_StateXXX constructs back to the 2.3.4 form and this seems to resolve the problem (I've attached the updated threadmodule.c files for both 2.3.5 and 2.4.1). ---------------------------------------------------------------------- Comment By: John Belmonte (jbelmonte) Date: 2005-04-16 21:27 Message: Logged In: YES user_id=282299 This problem also occurs on Linux. Note however that I've traced the source of the symptom to a different bug fix, namely [1010677] thread Module Breaks PyGILState_Ensure() (http://sourceforge.net/tracker/index.php?func=detail&aid=1010677&group_id=5470&atid=305470). Specifically, the change from v2.56 to v2.56.8.1 of threadmodule.c. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2005-04-16 20:21 Message: Logged In: YES user_id=357491 To answer the comment on bug #754449 and the email I got from someone, there is no way my fix for that bug would cause this. I only touched the 'threading' module, nothing else. That is in pure Python and thus has no direct way of modifying what does and does not execute in restricted mode. If I understand how restricted execution works that is all set at the C level. Plus the fix only deals with the teardown code; it in no way interfaces with how threads are executed. All changed code come after the thread and completed execution. The only thing it does ahead of time is store a reference to stdout. In other words it ain't my fault to the best of my knowledge. =) ---------------------------------------------------------------------- Comment By: Alan Davies (goatpunch) Date: 2005-03-31 07:24 Message: Logged In: YES user_id=967140 The same problem definately does occur with mod_python, I've found it to occur with Python 2.3.5 and 2.4 on Win32: http://www.modpython.org/pipermail/mod_python/2005- March/017802.html ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1163563&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
