Re: [Python-Dev] Bug tracker outage
On Wed, Mar 28, 2012 at 19:47, "Martin v. Löwis" wrote: > Am 27.03.2012 23:11, schrieb "Martin v. Löwis": >> Upfront hosting (Izak Burger) is going to do a Debian upgrade of the bug >> tracker machine "soon" (likely tomorrow). This may cause some outage, >> since there is a lot of custom stuff on the machine which may >> break with newer (Python) versions. I'll notify here when the upgrade >> is complete. > > The upgrade is complete. It was in fact the Postgres upgrade (to 8.4) > which caused the longest blackout time. Curiously, did the update have anything to do with Debian 5's EOL? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] possible distutils.sysconfig.get_config_var bug
Hi python-dev,
Sorry if this is the wrong place to discuss this potential bug - feel free to
point me in the right direction if so.
I'm running OS X 10.7.3, and have two python2.7s installed, one system default
in /usr/bin and the other by homebrew symlinked in /usr/local/bin.
While running a configure script, distutils.sysconfig.get_config_var is not
returing a full path for my homebrew Python framework, like so:
Python 2.7.2 (default, Mar 28 2012, 02:31:16)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on darwin
>>> distutils.sysconfig.get_config_var('LINKFORSHARED')
'-u _PyMac_Error Python.framework/Versions/2.7/Python'
(the full path should be
/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/Python)
Whereas from the system python in /usr/bin:
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
>>> distutils.sysconfig.get_config_var('LINKFORSHARED')
'-u _PyMac_Error
/System/Library/Frameworks/Python.framework/Versions/2.7/Python'
This seems like a bug, no?
Thanks for the help,
Jordan
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
Carl, I've drafted some notes: http://piratepad.net/PAZ3CEq9CZ Please feel free to edit them. If you want to chat, I can often be reached on freenode as 'jaraco' or XMPP at my e-mail address if you want to sprint on this in real-time. Does the issue only exist for Python 2.6 and 2.7? I'm not familiar with the release process. What's the next step? > -Original Message- > From: R. David Murray [mailto:[email protected]] > Sent: Wednesday, 28 March, 2012 17:46 > > I think it is reasonable to put something in the release notes. This change is > much larger than changes we normally make in maintenance release, > because it fixes a security bug. But because it is larger than normal, adding > release notes like this about known breakage is, I think, a good idea. > > Perhaps you and Carl could collaborate on a page explaining the issue in > detail, and on a brief note to include in the release notes that points to your > more extensive discussion? smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 418: Add monotonic clock
On Wed, Mar 28, 2012 at 23:40, Victor Stinner wrote: >> Does this primarily give a high resolution clock, or primarily a >> monotonic clock? That's not clear from either the name, or the PEP. > > I expect a better resolution from time.monotonic() than time.time(). Sure. And for me that means that time.hires() would give a high resolution version of time.time(). Ie, not monotonic, but wall clock. The question then is why time.time() doesn't give that resolution from the start. It seems to me we need three functions: One to get the wall clock, one to get a monotonic clock, and one that falls back if no monotonic clock is available. Both time.time() and time.monotonic() should give the highest resolution possible. As such, time.hires() seems pointless. //Lennart ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
On Thu, 29 Mar 2012 11:41:46 -, "Jason R. Coombs" wrote: > Does the issue only exist for Python 2.6 and 2.7? It might exist for 3.1 and 3.2 as well. > I'm not familiar with the release process. What's the next step? I would suggest opening an issue on the tracker and marking it as a release-blocker. That way the release managers will see it and can decide what if anything they want to do about it. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] possible distutils.sysconfig.get_config_var bug
If you could, Jordan, please file a bug at bugs.python.org so the
discussion can happen there and be tracked better.
On Wed, Mar 28, 2012 at 03:06, Jordan Schneider wrote:
> Hi python-dev,
>
> Sorry if this is the wrong place to discuss this potential bug - feel free
> to point me in the right direction if so.
>
> I'm running OS X 10.7.3, and have two python2.7s installed, one system
> default in /usr/bin and the other by homebrew symlinked in /usr/local/bin.
>
> While running a configure script, distutils.sysconfig.get_config_var is
> not returing a full path for my homebrew Python framework, like so:
>
> Python 2.7.2 (default, Mar 28 2012, 02:31:16)
> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on
> darwin
> >>> distutils.sysconfig.get_config_var('LINKFORSHARED')
> '-u _PyMac_Error Python.framework/Versions/2.7/Python'
>
> (the full path should be
> /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/Python)
>
>
> Whereas from the system python in /usr/bin:
>
> Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on
> darwin
> >>> distutils.sysconfig.get_config_var('LINKFORSHARED')
> '-u _PyMac_Error
> /System/Library/Frameworks/Python.framework/Versions/2.7/Python'
>
> This seems like a bug, no?
>
> Thanks for the help,
> Jordan
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug tracker offline again for re-indexing
On 03/29/2012 05:07 AM, "Martin v. Löwis" wrote: >>> I noticed that text search wasn't working right on the bug tracker, and >>> Martin >>> has taken it offline again to re-index. >> >> which will, unfortunately, take a few more hours to complete. > > It seems to work now, so I turned it on again. Text search now uses > Xapian, and recreating the Xapian index of all msg objects took a while. > Is the search working properly: Search all for "virtualenv" at the top right and issue 14357 doesn't appear in the list of results... Also, search open issues for "subprocess" and there is only 1 result. I wish :-) -- Ross Lagerwall signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug tracker offline again for re-indexing
On Thu, 29 Mar 2012 18:21:34 +0200, Ross Lagerwall wrote: > On 03/29/2012 05:07 AM, "Martin v. Löwis" wrote: > >>> I noticed that text search wasn't working right on the bug tracker, and > >>> Martin > >>> has taken it offline again to re-index. > >> > >> which will, unfortunately, take a few more hours to complete. > > > > It seems to work now, so I turned it on again. Text search now uses > > Xapian, and recreating the Xapian index of all msg objects took a while. > > > > Is the search working properly: Search all for "virtualenv" at the top > right and issue 14357 doesn't appear in the list of results... > > Also, search open issues for "subprocess" and there is only 1 result. I > wish :-) I get three for 'all' issues, which is certainly wrong. All of them have subprocess in the title. I suspect the search is only searching the title field, which is wrong. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
Thanks Jason for raising this. I just assumed that this was virtualenv's fault (which it is) and didn't consider raising it here, but a note in the release notes for the affected Python versions will certainly reach many more of the likely-to-be-affected users. FTR, I confirmed that the issue also affects the upcoming point releases for 3.1 and 3.2, as well as 2.6 and 2.7. Jason filed issue 1 to track the addition to the release notes for those versions. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
On Wed, 2012-03-28 at 18:22 +, Jason R. Coombs wrote: > I see this was reported as a debian bug. > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665776 > > To reproduce, using virtualenv 1.7+ on Python 2.7.2 on Ubuntu, create > a virtualenv. Move that virtualenv to a host with Python 2.7.3RC2 > yields: > jaraco@vdm-dev:~$ /usr/bin/python2.7 -V > Python 2.7.3rc2 > jaraco@vdm-dev:~$ env/bin/python -V > Python 2.7.2 > jaraco@vdm-dev:~$ env/bin/python -c "import os; os.urandom()" > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'urandom' It looks like this a symptom of the move of urandom to os.py to posximodule et al. At first glance, it looks like this specific hunk should be reverted: http://hg.python.org/cpython/rev/a0f43f4481e0#l7.1 so that if you're running with the new stdlib but an old python binary the combination can still have a usable os.urandom Should this be tracked in bugs.python.org? Hope this is helpful Dave ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug tracker offline again for re-indexing
On 03/29/2012 06:39 PM, R. David Murray wrote: > I get three for 'all' issues, which is certainly wrong. All of them > have subprocess in the title. > > I suspect the search is only searching the title field, which is wrong. No, #14357 contains 'virtualenv' in the title but it doesn't come up in the 'all' search. -- Ross Lagerwall signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
On 03/29/2012 11:39 AM, David Malcolm wrote: >> jaraco@vdm-dev:~$ env/bin/python -c "import os; os.urandom()" >> Traceback (most recent call last): >> File "", line 1, in >> AttributeError: 'module' object has no attribute 'urandom' > > It looks like this a symptom of the move of urandom to os.py to > posximodule et al. > > At first glance, it looks like this specific hunk should be reverted: > http://hg.python.org/cpython/rev/a0f43f4481e0#l7.1 > so that if you're running with the new stdlib but an old python binary > the combination can still have a usable os.urandom Indeed, I've just tested and verified that this does fix the problem. > Should this be tracked in bugs.python.org? I've added this option as a comment on bug 1. The title of that bug is worded such that it could be reasonably resolved either with the backwards-compatibility fix or the release notes addition, the release managers can decide what seems appropriate to them. Carl signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug tracker offline again for re-indexing
Am 29.03.2012 18:21, schrieb Ross Lagerwall: > On 03/29/2012 05:07 AM, "Martin v. Löwis" wrote: I noticed that text search wasn't working right on the bug tracker, and Martin has taken it offline again to re-index. >>> >>> which will, unfortunately, take a few more hours to complete. >> >> It seems to work now, so I turned it on again. Text search now uses >> Xapian, and recreating the Xapian index of all msg objects took a while. >> > > Is the search working properly: Search all for "virtualenv" at the top > right and issue 14357 doesn't appear in the list of results... > > Also, search open issues for "subprocess" and there is only 1 result. I > wish :-) Please submit an issue to the meta tracker. It may take weeks until I can get to it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in generator if the generator in created in a C thread
On Wed, Mar 28, 2012 at 06:45, Victor Stinner wrote:
> We have a crash in our product when tracing is enabled by
> sys.settrace() and threading.settrace(). If a Python generator is
> created in a C thread, calling the generator later in another thread
> may crash if Python tracing is enabled.
>
> - the C thread calls PyGILState_Ensure() which creates a temporary
> Python thread state
> - a generator is created, the generator has a reference to a Python
> frame which keeps a reference to the temporary Python thread state
> - the C thread calls PyGILState_Releases() which destroys the
> temporary Python thread state
> - when the generator is called later in another thread, call_trace()
> reads the Python thread state from the generator frame, which is the
> destroyed frame => it does crash on a pointer dereference if the
> memory was reallocated (by malloc()) and the data were erased
>
This is timely. We've seen several similar bugs in our 3.1.2-based
embedded product--in fact, I'm tracking down what might be another
this week.
The common theme is that C extension code that depends on
PyGILState_Ensure() for GIL management runs the risk of causing latent
crashes in any situation that involves preserving a frame beyond the
lifetime of the thread that created it.
In our case, the culprits have usually been the frames attached to
exceptions thrown across the extension->Python->embedding app
boundaries.
From a theoretical standpoint, I can't quite decide what the real error is:
1) the fact that PyGILState_Release() destroys a temporary thread
state that may still be referenced by some objects, or
2) the fact that some code is trying to keep frame objects after the
creating thread state no longer exists.
This week I've been leaning toward 2), but then I realized that
keeping frames post-thread-death is not that uncommon (for example,
debuggers and other diagnostic techniques like
http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html).
Locally we added some (unfortunate) code to our 3.1.2 port to wrap
PyGILState_Ensure(), which I thought had sidestepped the issue for us:
void takeGIL()
{
PyGILState_Ensure();
// This has the side effect of keeping such thread states alive until
// the interpreter is finalized; however, all thread state objects get
// unconditionally deleted during Py_Finalize, so they won't leak.
PyThreadState* pThreadState = PyGILState_GetThisThreadState();
if (pThreadState->gilstate_counter == 1)
{
++pThreadState->gilstate_counter;
}
}
But clearly that can't be a correct answer (and it may not even be a
functioning one, given that I'm seeing a similar issue again).
--
Tim Lesher
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issue 14417: consequences of new dict runtime error
Some of us have expressed uneasiness about the consequences of dict raising an error on lookup if the dict has been modified, the fix Victor made to solve one of the crashers. I don't know if I speak for the others, but (assuming that I understand the change correctly) my concern is that there is probably a significant amount of threading code out there that assumes that dict *lookup* is a thread-safe operation. Much of that code will, if moved to Python 3.3, now be subject to random runtime errors for which it will not be prepared. Further, code which appears safe can suddenly become unsafe if a refactoring of the code causes an object to be stored in the dictionary that has a Python equality method. Would it be possible to modify the fix so that the lookup is retried a non-trivial but finite number of times, so that normal code will work and only pathological code will break? I know that I really don't want to think about having to audit the (significantly threaded) application I'm currently working on to make sure it is "3.3 safe". Dict lookup operations are *common*, and we've never had to think about whether or not they were thread-safe before (unless there were inter-thread synchronization issues involved, of course). Nor am I sure the locking dict type suggested by Jim on the issue would help, since a number of the dicts we are using are produced by library code. So we'd have to wait for those libraries to be ported to 3.3 --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug tracker offline again for re-indexing
On Thu, 29 Mar 2012 21:14:32 +0200, =?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > Am 29.03.2012 18:21, schrieb Ross Lagerwall: > > On 03/29/2012 05:07 AM, "Martin v. Löwis" wrote: > I noticed that text search wasn't working right on the bug tracker, and > Martin > has taken it offline again to re-index. > >>> > >>> which will, unfortunately, take a few more hours to complete. > >> > >> It seems to work now, so I turned it on again. Text search now uses > >> Xapian, and recreating the Xapian index of all msg objects took a while. > >> > > > > Is the search working properly: Search all for "virtualenv" at the top > > right and issue 14357 doesn't appear in the list of results... > > > > Also, search open issues for "subprocess" and there is only 1 result. I > > wish :-) > > Please submit an issue to the meta tracker. It may take weeks until I > can get to it. I've reopened issue 443, with an example of a failing search giving relevant issue numbers. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > Some of us have expressed uneasiness about the consequences of dict > raising an error on lookup if the dict has been modified, the fix Victor > made to solve one of the crashers. > > I don't know if I speak for the others, but (assuming that I understand > the change correctly) my concern is that there is probably a significant > amount of threading code out there that assumes that dict *lookup* is > a thread-safe operation. Much of that code will, if moved to Python > 3.3, now be subject to random runtime errors for which it will not > be prepared. Further, code which appears safe can suddenly become > unsafe if a refactoring of the code causes an object to be stored in > the dictionary that has a Python equality method. My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. > Would it be possible to modify the fix so that the lookup is retried a > non-trivial but finite number of times, so that normal code will work > and only pathological code will break? FWIW a similar approach was rejected as a fix for the hash DoS attack. > I know that I really don't want to think about having to audit the > (significantly threaded) application I'm currently working on to make sure > it is "3.3 safe". Dict lookup operations are *common*, and we've never > had to think about whether or not they were thread-safe before (unless > there were inter-thread synchronization issues involved, of course). > Nor am I sure the locking dict type suggested by Jim on the issue would > help, since a number of the dicts we are using are produced by library > code. So we'd have to wait for those libraries to be ported to 3.3 Agreed that this is somewhat scary. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray > wrote: > > Some of us have expressed uneasiness about the consequences of dict > > raising an error on lookup if the dict has been modified, the fix Victor > > made to solve one of the crashers. > > > > I don't know if I speak for the others, but (assuming that I understand > > the change correctly) my concern is that there is probably a significant > > amount of threading code out there that assumes that dict *lookup* is > > a thread-safe operation. Â Much of that code will, if moved to Python > > 3.3, now be subject to random runtime errors for which it will not > > be prepared. Â Further, code which appears safe can suddenly become > > unsafe if a refactoring of the code causes an object to be stored in > > the dictionary that has a Python equality method. > > My original assessment was that this only affects dicts whose keys > have a user-implemented __hash__ or __eq__ implementation, and that > the number of apps that use this *and* assume the threadsafe property > would be pretty small. This is just intuition, I don't have hard > facts. But I do want to stress that not all dict lookups automatically > become thread-unsafe, only those that need to run user code as part of > the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. Are there any other places in Python where substituting a duck-typed Python class or a Python subclass can cause a runtime error in previously working code? > > Would it be possible to modify the fix so that the lookup is retried a > > non-trivial but finite number of times, so that normal code will work > > and only pathological code will break? > > FWIW a similar approach was rejected as a fix for the hash DoS attack. Yes, but in this case the non-counting version breaks just as randomly, but more often. So arguing that counting here is analogous to counting in the DoS attack issue is an argument for removing the fix entirely :) The counting version could use a large enough count (since the count used to be infinite!) that only code that would be having pathological performance anyway would raise the runtime error, rather than any code (that uses python __eq__ on keys) randomly raising a runtime error, which is what we have now. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On Thu, 29 Mar 2012 16:31:03 -0400, "R. David Murray" wrote: > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > > My original assessment was that this only affects dicts whose keys > > have a user-implemented __hash__ or __eq__ implementation, and that > > the number of apps that use this *and* assume the threadsafe property > > would be pretty small. This is just intuition, I don't have hard > > facts. But I do want to stress that not all dict lookups automatically > > become thread-unsafe, only those that need to run user code as part of > > the key lookup. > > You are probably correct, but the thing is that one still has to do the > code audit to be sure...and then make sure that no one later introduces > such an object type as a dict key. I just did a quick grep on our project. We are only defining __eq__ and __hash__ a couple places, but both are objects that could easily get used as dict keys (there is a good chance that's *why* those methods are defined) accessed by more than one thread. I haven't done the audit to find out :) The libraries we depend on have many more definitions of __eq__ and __hash__, and we'd have to check them too. (Including SQLAlchemy, and I wouldn't want that job.) So our intuition that this is not common may be wrong. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
R. David Murray, 29.03.2012 22:31: > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: >> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: >>> Some of us have expressed uneasiness about the consequences of dict >>> raising an error on lookup if the dict has been modified, the fix Victor >>> made to solve one of the crashers. >>> >>> I don't know if I speak for the others, but (assuming that I understand >>> the change correctly) my concern is that there is probably a significant >>> amount of threading code out there that assumes that dict *lookup* is >>> a thread-safe operation. Much of that code will, if moved to Python >>> 3.3, now be subject to random runtime errors for which it will not >>> be prepared. Further, code which appears safe can suddenly become >>> unsafe if a refactoring of the code causes an object to be stored in >>> the dictionary that has a Python equality method. >> >> My original assessment was that this only affects dicts whose keys >> have a user-implemented __hash__ or __eq__ implementation, and that >> the number of apps that use this *and* assume the threadsafe property >> would be pretty small. This is just intuition, I don't have hard >> facts. But I do want to stress that not all dict lookups automatically >> become thread-unsafe, only those that need to run user code as part of >> the key lookup. > > You are probably correct, but the thing is that one still has to do the > code audit to be sure...and then make sure that no one later introduces > such an object type as a dict key. The thing is: the assumption that arbitrary dict lookups are GIL-atomic has *always* been false. Only those that do not involve Python code execution for the hash key calculation or the object comparison are. That includes the built-in strings and numbers (and tuples of them), which are by far the most common dict keys. Looking up arbitrary user provided objects is definitely not guaranteed to be atomic. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/29/2012 04:48 PM, R. David Murray wrote: On Thu, 29 Mar 2012 16:31:03 -0400, "R. David Murray" wrote: On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. I just did a quick grep on our project. We are only defining __eq__ and __hash__ a couple places, but both are objects that could easily get used as dict keys (there is a good chance that's *why* those methods are defined) accessed by more than one thread. I haven't done the audit to find out :) The libraries we depend on have many more definitions of __eq__ and __hash__, and we'd have to check them too. (Including SQLAlchemy, and I wouldn't want that job.) So our intuition that this is not common may be wrong. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com Hm, as far I understand this seems like an issue for gnu PTH, not python job, which should transparently handles thread safety issues based on the host/machine capabilities. Therefore I hope the fix in python don't affect thread-unsafe apps to raise spurious RuntimeErrors when a dict get modified across a SMP-aware platform... :-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in generator if the generator in created in a C thread
On 29/03/2012 21:35, Tim Lesher wrote:
From a theoretical standpoint, I can't quite decide what the real error is:
1) the fact that PyGILState_Release() destroys a temporary thread
state that may still be referenced by some objects, or
2) the fact that some code is trying to keep frame objects after the
creating thread state no longer exists.
This week I've been leaning toward 2), but then I realized that
keeping frames post-thread-death is not that uncommon (for example,
debuggers and other diagnostic techniques like
http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html).
The problem is not the frame, but the Python thread state referenced by
the frame. It's a private attribute. My patch just updates this
reference before running the generator (and it clears the reference when
the generator excution is stopped or finished).
Locally we added some (unfortunate) code to our 3.1.2 port to wrap
PyGILState_Ensure(), which I thought had sidestepped the issue for us:
void takeGIL()
{
PyGILState_Ensure();
// This has the side effect of keeping such thread states alive until
// the interpreter is finalized; however, all thread state objects get
// unconditionally deleted during Py_Finalize, so they won't leak.
PyThreadState* pThreadState = PyGILState_GetThisThreadState();
if (pThreadState->gilstate_counter == 1)
{
++pThreadState->gilstate_counter;
}
}
But clearly that can't be a correct answer (and it may not even be a
functioning one, given that I'm seeing a similar issue again).
You may leak memory if your threads have a short lifetime and you create
many threads. For example if one thread is only used to process one
request and then is destroyed.
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel wrote: > R. David Murray, 29.03.2012 22:31: > > On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: > >> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: > >>> Some of us have expressed uneasiness about the consequences of dict > >>> raising an error on lookup if the dict has been modified, the fix Victor > >>> made to solve one of the crashers. > >>> > >>> I don't know if I speak for the others, but (assuming that I understand > >>> the change correctly) my concern is that there is probably a significant > >>> amount of threading code out there that assumes that dict *lookup* is > >>> a thread-safe operation. Much of that code will, if moved to Python > >>> 3.3, now be subject to random runtime errors for which it will not > >>> be prepared. Further, code which appears safe can suddenly become > >>> unsafe if a refactoring of the code causes an object to be stored in > >>> the dictionary that has a Python equality method. > >> > >> My original assessment was that this only affects dicts whose keys > >> have a user-implemented __hash__ or __eq__ implementation, and that > >> the number of apps that use this *and* assume the threadsafe property > >> would be pretty small. This is just intuition, I don't have hard > >> facts. But I do want to stress that not all dict lookups automatically > >> become thread-unsafe, only those that need to run user code as part of > >> the key lookup. > > > > You are probably correct, but the thing is that one still has to do the > > code audit to be sure...and then make sure that no one later introduces > > such an object type as a dict key. > > The thing is: the assumption that arbitrary dict lookups are GIL-atomic has > *always* been false. Only those that do not involve Python code execution > for the hash key calculation or the object comparison are. That includes > the built-in strings and numbers (and tuples of them), which are by far the > most common dict keys. Looking up arbitrary user provided objects is > definitely not guaranteed to be atomic. Well, I'm afraid I was using the term 'thread safety' rather too loosely there. What I mean is that if you do a dict lookup, the lookup either returns a value or a KeyError, and that if you get back an object that object has internally consistent state. The problem this fix introduces is that the lookup may fail with a RuntimeError rather than a KeyError, which it has never done before. I think that is what Guido means by code that uses objects with python eq/hash *and* assumes threadsafe lookup. If mutation of the objects or dict during the lookup is a concern, then the code would use locks and wouldn't have the problem. But there are certainly situations where it doesn't matter if the dictionary mutates during the lookup, as long as you get either an object or a KeyError, and thus no locks are (currently) needed. Maybe I'm being paranoid about breakage here, but as with most backward compatibility concerns, there are probably more bits of code that will be affected than our intuition indicates. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Integrating the PEP 397 launcher
After talking with Martin and several others during the language summit and elsewhere around PyCon, PEP 397 should be accepted. I don't remember who, but some suggested it should just be a regular old feature instead of going through the PEP process. So...does this even need to continue the PEP process? Vinay - is the code you have on bitbucket ready to roll into CPython, thus into the installer? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
2012/3/29 Brian Curtin : > After talking with Martin and several others during the language > summit and elsewhere around PyCon, PEP 397 should be accepted. I don't > remember who, but some suggested it should just be a regular old > feature instead of going through the PEP process. So...does this even > need to continue the PEP process? If you have a PEP and it's accepted, what would be the difference? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: > 2012/3/29 Brian Curtin : >> After talking with Martin and several others during the language >> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >> remember who, but some suggested it should just be a regular old >> feature instead of going through the PEP process. So...does this even >> need to continue the PEP process? > > If you have a PEP and it's accepted, what would be the difference? In the end? Nothing. It was a comment about this whole process not even needing to exist. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
On 3/29/2012 3:50 PM, Brian Curtin wrote: On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: > 2012/3/29 Brian Curtin: >> After talking with Martin and several others during the language >> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >> remember who, but some suggested it should just be a regular old >> feature instead of going through the PEP process. So...does this even >> need to continue the PEP process? > > If you have a PEP and it's accepted, what would be the difference? In the end? Nothing. It was a comment about this whole process not even needing to exist. It was pretty controversial when it started... but it is such obviously beneficial functionality... and works well... All it needs is official acceptance now, and integration into the release, no? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
On Thu, Mar 29, 2012 at 18:08, Glenn Linderman wrote: > All it needs is official acceptance now, and integration into the release, > no? If it wasn't clear, this is what I said in the first post. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
2012/3/29 Brian Curtin : > On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: >> 2012/3/29 Brian Curtin : >>> After talking with Martin and several others during the language >>> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >>> remember who, but some suggested it should just be a regular old >>> feature instead of going through the PEP process. So...does this even >>> need to continue the PEP process? >> >> If you have a PEP and it's accepted, what would be the difference? > > In the end? Nothing. It was a comment about this whole process not > even needing to exist. The PEP process in general or specifically for this PEP? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
Brian Curtin python.org> writes: > Vinay - is the code you have on bitbucket ready to roll into CPython, > thus into the installer? I believe the main C launcher code is ready to roll into CPython. However, the standalone installer I provide uses WiX rather than msilib, and includes additional executables for functionality like "associate .py with one of the installed Pythons" when the launcher is uninstalled, and for printing messages in certain contexts when installing. I believe there needs to be a little more thought given to how to bring the launcher into the main installer to see if we can either dispense with, or make suitable changes to, these ancillary functions. I would appreciate some feedback from Martin about this - as far as I know he has not made any comments about launcher integration into the main installer. The current launcher functionality (py[w].exe) is as outlined in the PEP + feedback from users (e.g. your recent suggestion to use LOCALAPPDATA rather than APPDATA). The test harness may also need some thinking about - as the launcher executable is separate from Python, I'm not sure if it's appropriate just to create a "test_launcher.py" in Lib/test. To do a full test of the launcher you need multiple 2.x and 3.x versions installed, and I'm not sure if this could be done on existing Windows buildbots, for example. Of course it could be done with mocked executables and synthetically-added registry entries, but that isn't currently in place. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
On Fri, Mar 30, 2012 at 7:50 AM, Brian Curtin wrote: > On Thu, Mar 29, 2012 at 17:45, Benjamin Peterson wrote: >> 2012/3/29 Brian Curtin : >>> After talking with Martin and several others during the language >>> summit and elsewhere around PyCon, PEP 397 should be accepted. I don't >>> remember who, but some suggested it should just be a regular old >>> feature instead of going through the PEP process. So...does this even >>> need to continue the PEP process? >> >> If you have a PEP and it's accepted, what would be the difference? > > In the end? Nothing. It was a comment about this whole process not > even needing to exist. Hindsight, as usual, is 20-20. If the process got enough discussion that somebody said, "look, this really needs a PEP," it's worth the effort to record it. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
On Fri, Mar 30, 2012 at 4:26 AM, Carl Meyer wrote: > I've added this option as a comment on bug 1. The title of that bug > is worded such that it could be reasonably resolved either with the > backwards-compatibility fix or the release notes addition, the release > managers can decide what seems appropriate to them. +1 for restoring the fallback code in os.py. A security release definitely shouldn't be breaking that kind of thing. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
Le 29/03/2012 22:04, Nick Coghlan a écrit : > On Fri, Mar 30, 2012 at 4:26 AM, Carl Meyer wrote: >> I've added this option as a comment on bug 1. The title of that bug >> is worded such that it could be reasonably resolved either with the >> backwards-compatibility fix or the release notes addition, the release >> managers can decide what seems appropriate to them. > +1 for restoring the fallback code in os.py. A security release > definitely shouldn't be breaking that kind of thing. The RMs have already agreed not to restore the fallback, and the maintainer of virtualenv agreed. See report for more details. Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
On Fri, Mar 30, 2012 at 12:07 PM, Éric Araujo wrote: > Le 29/03/2012 22:04, Nick Coghlan a écrit : >> On Fri, Mar 30, 2012 at 4:26 AM, Carl Meyer wrote: >>> I've added this option as a comment on bug 1. The title of that bug >>> is worded such that it could be reasonably resolved either with the >>> backwards-compatibility fix or the release notes addition, the release >>> managers can decide what seems appropriate to them. >> +1 for restoring the fallback code in os.py. A security release >> definitely shouldn't be breaking that kind of thing. > > The RMs have already agreed not to restore the fallback, and the > maintainer of virtualenv agreed. See report for more details. The details are pretty short (and make sense), so I'll repeat the most salient points here for the benefit of others: - if the binary in a virtualenv isn't updated, then projects running in a virtualenv won't receive the security fix when the updated version of Python is installed - restoring the fallback in os.py would make this failure mode *silent*, so a user could upgrade Python, set PYTHONHASHSEED, but fail to realise they also need to update the binary in the virtualenv in order to benefit from the hash protection - with the current behaviour, failing to upgrade the binary results in a noisy ImportError or AttributeError related to os.urandom, which the release notes and virtualenv help channels can instruct people on handling. This approach explicitly covers the additional steps needed to fully deploy the security fix when using virtualenv. That rationale makes sense to me, too. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default
On Fri, Mar 30, 2012 at 2:01 AM, andrew.svetlov wrote: > +- Issue #14409: IDLE doesn't not execute commands from shell, > + error with default keybinding for Return. (Patch by Roger Serwy) The double negative here makes this impossible to understand. Could we please get an updated NEWS entry that explains what actually changed in IDLE to fix this? Perhaps something like "IDLE now always sets the default keybind for Return correctly, ensuring commands can be executed in the IDLE shell window"? (assuming that's what happened). This is important, folks: NEWS entries need to be comprehensible for people that *haven't* read the associated tracker issue. This means that issue titles (which generally describe a problem someone was having) are often inappropriate as NEWS items. NEWS items should be short descriptions that clearly describe *what changed*, perhaps with some additional information to explain a bit about why the change was made. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
Zitat von Brian Curtin : After talking with Martin and several others during the language summit and elsewhere around PyCon, PEP 397 should be accepted. I don't remember who, but some suggested it should just be a regular old feature instead of going through the PEP process. So...does this even need to continue the PEP process? I don't think PEP 397 can be accepted as it stands; it has too many open issues. However, I am in favor of accepting the proposed feature. Now that we do have the PEP, I think that should be done properly. I thought you offered to rewrite it. Formally, I could accept the PEP being withdrawn, and the feature integrated anyway, but I still consider that bad style. So: I can offer to rewrite the PEP to give a full specification of the feature. It might be that I still need some help to provide end-user prose in the PEP if people would want to see that as well. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
On Thu, Mar 29, 2012 at 22:48, wrote: > Now that we do have the PEP, I think that should be done properly. > I thought you offered to rewrite it. There are definitely areas that I would like to work on, especially pulling implementation details out and replacing with, as you say, end-user prose. For example, some part of a doc tells you to call some function with a specific parameter to figure out where py.ini should be found - it needs to be replaced with an example directory. > So: I can offer to rewrite the PEP to give a full specification > of the feature. It might be that I still need some help to > provide end-user prose in the PEP if people would want to see that as > well. I would be much better at proposing the end-user stuff than the specification. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default
On Fri, Mar 30, 2012 at 2:12 PM, wrote: > > Zitat von Nick Coghlan : > > >> On Fri, Mar 30, 2012 at 2:01 AM, andrew.svetlov >> wrote: >>> >>> +- Issue #14409: IDLE doesn't not execute commands from shell, >>> + error with default keybinding for Return. (Patch by Roger Serwy) >> >> >> The double negative here makes this impossible to understand. Could we >> please get an updated NEWS entry that explains what actually changed >> in IDLE to fix this? > > > Please consider that Andrew is not a native speaker of English. So > it's unfair to ask him to rewrite the NEWS entry. That can only > be done by a native speaker. The NEWS entries need to be treated as being at least close to on par with the rest of the docs - it's OK if someone can't come up with the words themselves, but if that's the case, then it's preferable to ask for help with the wording explicitly. Is my suggested rephrasing correct? I don't know, as I'm not familiar with either the original problem or what was done to fix it. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
