Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
Guido van Rossum wrote: On Thu, Mar 8, 2012 at 4:33 PM, Nick Coghlan wrote: On Fri, Mar 9, 2012 at 3:31 AM, Guido van Rossum wrote: But the __builtins__ that are actually used by any particular piece of code is *not* taken by importing builtins. It is taken from what the globals store under the key __builtins__. This is a feature that was added specifically for sandboxing purposes, but I believe it has found other uses too. Agreed, but swapping out builtins for a different namespace is still the exception rather than the rule. My Impression of Mark's proposal was that this approach would become the *preferred* way of doing things, and that's the part I don't like at a conceptual level. The key point is that every piece of code already inherits locals, globals and builtins from somewhere else. We can already control locals (by which parameters are passed in) and globals via exec, eval, __import__, and runpy (any others?) but we can't control builtins. Correct - because controlling builtins is the domain of sandboxes. Incorrect (unless I misunderstand the context) -- when you control the globals you control the __builtins__ set there. And this is where I don't like the idea at a practical level. We already have a way to swap in a different set of builtins for a certain execution context (i.e. set "__builtins__" in the global namespace) for a small chunk of code, as well as allowing collections.ChainMap to insert additional namespaces into the name lookup path. This proposal suggests adding an additional mapping argument to every API that currently accepts a locals and/or globals mapping, thus achieving... well, nothing substantial, as far as I can tell (aside from a lot of pointless churn in a bunch of APIs, not all of which are under our direct control). In any case, the locals / globals / builtins chain is a simplification; there are also any number of intermediate scopes (between locals and globals) from which "nonlocal" variables may be used. Like optimized function globals, these don't use a dict lookup at all, they are determined by compile-time analysis. Acknowledged, but code executed via the exec API with both locals and globals passed in is actually one of the few places where that lookup chain survives in its original form (module level class definitions being the other). Now, rereading Mark's original message, a simpler proposal of having *function objects* do an early lookup of "self.__globals__['__builtins__']" at creation time and caching that somewhere such that the frame objects can get hold of it (rather than having to do the lookup every time the function gets called or a builtin gets referenced) might be a nice micro-optimisation. It's the gratuitous API changes that I'm objecting to, not the underlying idea of binding the reference to the builtins namespace earlier in the function definition process. I'd even be OK with leaving the default builtins reference *out* of the globals namespace in favour of storing a hidden reference on the frame objects. Agreed on the gratuitous API changes. I'd like to hear Mark's response. C API or Python API? The Python API would be changed, but in a backwards compatible way. exec, eval and __import__ would all gain an optional (keyword-only?) "builtins" parameter. I see no reason to change any of the C API functions. New functions taking an extra parameter could be added, but it wouldn't be a requirement. Cheers, Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - some numbers from the Django port
- Original Message - > But the stuff you run is not really benchmarking anything. As far as I > know django benchmarks benchmark something like mostly DB creation and > deletion, although that might differ between CPython and PyPy. How > about running *actual* django benchmarks, instead of the test suite? > > Not that proving anything is necessary, but if you try to prove > something, make it right. But my point was only to show that in a reasonable body of Python code (as opposed to a microbenchmark), the overhead of using wrappers was not significant. All those wrapper calls in ported Django and its test suite were exercised. It was not exactly a benchmarking exercise in that it didn't matter what the actual numbers were, nor was any claim being made about absolute performance; just that they were the same for all three variants, within statistical variation. As I mentioned in my other post, I happened to have the Django test suite figures to hand, and to my mind they suited the purpose of showing that wrapper calls, in the overall mix, don't seem to have a noticeable impact (whereas they do, in a microbenchmark). Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On Fri, Mar 9, 2012 at 6:19 PM, Mark Shannon wrote: > The Python API would be changed, but in a backwards compatible way. > exec, eval and __import__ would all gain an optional (keyword-only?) > "builtins" parameter. No, some APIs effectively define *protocols*. For such APIs, *adding* parameters is almost of comparable import to taking them away, because they require that other APIs modelled on the prototype also change. In this case, not only exec() has to change, but eval, __import__, probably runpy, function creation, eventually any third party APIs for code execution, etc, etc. Adding a new parameter to exec is a change with serious implications, and utterly unnecessary, since the API part is already covered by setting __builtins__ in the passed in globals namespace (which is appropriately awkward to advise people that they're doing something strange with potentially unintended consequences or surprising limitations). That said, binding a reference to the builtin *early* (for example, at function definition time or when a new invocation of the eval loop first fires up) may be a reasonable idea, but you don't have to change the user facing API to explore that option - it works just as well with "__builtins__" as an optional value in the existing global namespace. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
Nick Coghlan wrote:
On Fri, Mar 9, 2012 at 6:19 PM, Mark Shannon wrote:
The Python API would be changed, but in a backwards compatible way.
exec, eval and __import__ would all gain an optional (keyword-only?)
"builtins" parameter.
No, some APIs effectively define *protocols*. For such APIs, *adding*
parameters is almost of comparable import to taking them away, because
they require that other APIs modelled on the prototype also change. In
this case, not only exec() has to change, but eval, __import__,
probably runpy, function creation, eventually any third party APIs for
code execution, etc, etc.
Adding a new parameter to exec is a change with serious implications,
and utterly unnecessary, since the API part is already covered by
setting __builtins__ in the passed in globals namespace (which is
appropriately awkward to advise people that they're doing something
strange with potentially unintended consequences or surprising
limitations).
It is the implementation that interests me.
Implementing the (locals, globals, builtins) triple as a single object
has advantages both in terms of internal consistency and efficiency.
I just thought to expose this to the user.
I am now persuaded that I don't want to expose anything :)
That said, binding a reference to the builtin *early* (for example, at
function definition time or when a new invocation of the eval loop
first fires up) may be a reasonable idea, but you don't have to change
the user facing API to explore that option - it works just as well
with "__builtins__" as an optional value in the existing global
namespace.
OK. So, how about this:
(builtins refers to the dict used for variable lookup, not the module)
New eval pseudocode
eval(code, globals, locals):
triple = (locals, globals, globals["__builtins__"])
return eval_internal(triple)
Similarly for exec, __import__ and runpy.
That way the (IMO clumsy) builtins = globals["__builtins__"]
only happens at a few known locations.
It should then be clear where all code gets its namespaces from.
Namespaces should be inherited as follows:
frame:
function scope: globals and builtins from function, locals from parameters.
module scope: globals and builtins from module, locals == globals.
in eval, exec, or runpy: all explicit.
function: globals and builtins from module (no locals)
module: globals and builtins from import (no locals)
import: explicitly from __import__() or
implicitly from current frame in an import statement.
For frame and function, free and cell (nonlocal) variables would be
unchanged.
On entry the namespaces will be {}, {}, sys.modules['builtins'].__dict__
This is pretty much what happens anyway,
except that where code gets its builtins from is now well defined.
Cheers,
Mark.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] FYI dict lookup now raises a RuntimeError if the dict is modified during the lookup
If you use your own comparaison function (__eq__) for objects used as dict keys, you may now get RuntimeError with Python 3.3 if the dict is modified during a dict lookup in a multithreaded application. You should use a lock on the dict to avoid this exception. Said differently, a dict lookup is not atomic if you use your own types as dict keys and these types implement __eq__. In Python < 3.3, the dict lookup is retried until the dict is no more modified during the lookup, which leads to a stack overflow crash if the dict is always modified. See the issue #14205 for the rationale. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
> The reason I am proposing this here rather than on python-ideas is that > treating the triple of [locals, globals, builtins] as a single > "execution context" can be implemented in a really nice way. > > Internally, the execution context of [locals, globals, builtins] > can be treated a single immutable object (custom object or tuple) > Treating it as immutable means that it can be copied merely by taking a > reference. A nice trick in the implementation is to make a NULL locals > mean "fast" locals for function contexts. Frames, could then acquire their > globals and builtins by a single reference copy from the function object, > rather than searching globals for a '__builtins__' > to find the builtins. Creating a new frame lookup for __builtins__ in globals only if globals of the new frame is different from the globals of the previous frame. You would like to optimize this case? If globals is unchanged, Python just increments the reference counter. When globals is different from the previous frame? When you call a function from a different module maybe? Do you have an idea of the speedup of your optimization? Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
Victor Stinner wrote: The reason I am proposing this here rather than on python-ideas is that treating the triple of [locals, globals, builtins] as a single "execution context" can be implemented in a really nice way. Internally, the execution context of [locals, globals, builtins] can be treated a single immutable object (custom object or tuple) Treating it as immutable means that it can be copied merely by taking a reference. A nice trick in the implementation is to make a NULL locals mean "fast" locals for function contexts. Frames, could then acquire their globals and builtins by a single reference copy from the function object, rather than searching globals for a '__builtins__' to find the builtins. Creating a new frame lookup for __builtins__ in globals only if globals of the new frame is different from the globals of the previous frame. You would like to optimize this case? If globals is unchanged, Python just increments the reference counter. I'm more interested in simplifying the code than performance. We this proposed approach, there is no need to test where the globals come from, or what the builtins are; just incref the namespace triple. When globals is different from the previous frame? When you call a function from a different module maybe? Do you have an idea of the speedup of your optimization? No. But it won't be slower. Cheers, Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a builtins parameter to eval(), exec() and __import__().
On 09/03/2012 12:57, Mark Shannon wrote: No. But it won't be slower. Cheers, Mark Please prove it, you have to convince a number of core developers including, but not limited to, the BDFL :). -- Cheers. Mark Lawrence. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2012-03-02 - 2012-03-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3318 (+19) closed 22704 (+44) total 26022 (+63) Open issues with patches: 1409 Issues opened (45) == #14179: Test coverage for lib/re.py http://bugs.python.org/issue14179 opened by flomm #14180: Factorize code to convert int/float to time_t, timeval or time http://bugs.python.org/issue14180 opened by haypo #14182: collections.Counter equality test thrown-off by zero counts http://bugs.python.org/issue14182 opened by rhettinger #14183: Test coverage for packaging.install and packaging.pypi.wrapper http://bugs.python.org/issue14183 opened by francismb #14184: test_recursion_limit fails on OS X when compiled with clang http://bugs.python.org/issue14184 opened by dk #14185: Failure to build _dbm with ndbm on Arch Linux http://bugs.python.org/issue14185 opened by anikom15 #14186: Link to PEP 3107 in "def" part of Language Reference http://bugs.python.org/issue14186 opened by cvrebert #14187: add "annotation" entry to Glossary http://bugs.python.org/issue14187 opened by cvrebert #14189: Documentation for some C APIs is missing clear specification o http://bugs.python.org/issue14189 opened by baruch.sterin #14190: Minor C API documentation bugs http://bugs.python.org/issue14190 opened by baruch.sterin #14191: argparse: nargs='*' doesn't get out-of-order positional parame http://bugs.python.org/issue14191 opened by v+python #14196: Unhandled exceptions in pdb return value display http://bugs.python.org/issue14196 opened by Simon.Chopin #14197: OS X framework builds do not create ABI-suffixed libpython3.x http://bugs.python.org/issue14197 opened by shibingli #14198: Backport parts of the new memoryview documentation http://bugs.python.org/issue14198 opened by skrah #14200: Idle shell crash on printing non-BMP unicode character http://bugs.python.org/issue14200 opened by vbr #14201: Documented caching for shared library's __getattr__ and __geti http://bugs.python.org/issue14201 opened by erijo #14202: The docs of xml.dom.pulldom are almost nonexistent http://bugs.python.org/issue14202 opened by eli.bendersky #14203: bytearray_getbuffer: unnecessary code http://bugs.python.org/issue14203 opened by skrah #14204: Support for the NPN extension to TLS/SSL http://bugs.python.org/issue14204 opened by colinmarc #14206: multiprocessing.Queue documentation is lacking important detai http://bugs.python.org/issue14206 opened by Garrett.Moore #14207: ElementTree.ParseError - needs documentation and consistent C& http://bugs.python.org/issue14207 opened by eli.bendersky #14208: No way to recover original argv with python -m http://bugs.python.org/issue14208 opened by Ben.Darnell #14209: pkgutil.iter_zipimport_modules ignores the prefix parameter fo http://bugs.python.org/issue14209 opened by James.Pickering #14210: add filename completion to pdb http://bugs.python.org/issue14210 opened by tshepang #14214: test_concurrent_futures hangs http://bugs.python.org/issue14214 opened by tebeka #14215: http://www.python.org/dev/peps/ title is python.org http://bugs.python.org/issue14215 opened by ramchandra.apte #14216: ImportError: No module named binascii http://bugs.python.org/issue14216 opened by liuqianhn #14219: start the Class tutorial in a more gentle manner http://bugs.python.org/issue14219 opened by tshepang #14222: Using time.time() in Queue.get breaks when system time is chan http://bugs.python.org/issue14222 opened by tasslehoff #14224: packaging: path description of resources is mixed up http://bugs.python.org/issue14224 opened by jokoala #14225: _cursesmodule compile error in OS X 32-bit-only installer buil http://bugs.python.org/issue14225 opened by ned.deily #14226: Expose dict_proxy type from descrobject.c http://bugs.python.org/issue14226 opened by ndparker #14227: console w/ cp65001 displays extra characters for non-ascii str http://bugs.python.org/issue14227 opened by metolone #14228: It is impossible to catch sigint on startup in python code http://bugs.python.org/issue14228 opened by telmich #14229: On KeyboardInterrupt, the exit code should mirror the signal n http://bugs.python.org/issue14229 opened by pitrou #14230: Delegating generator is not always visible to debugging tools http://bugs.python.org/issue14230 opened by Mark.Shannon #14232: obmalloc: mmap() returns MAP_FAILED on error, not 0 http://bugs.python.org/issue14232 opened by haypo #14234: CVE-2012-0876 (hash table collisions CPU usage DoS) for embedd http://bugs.python.org/issue14234 opened by dmalcolm #14235: test_cmd.py does not correctly call reload() http://bugs.python.org/issue14235 opened by Josh.Watson #14236: In help(re) are insufficient details http://bugs.python.org/issue14236 opened by py.user #14237: Special seq
[Python-Dev] Testsuite dependency on _testcapi
While testing Python 2.7 internally (at Google) I noticed that (now that ImportErrors aren't automatically test skips) lots of tests fail if you don't have the _testcapi module. These tests are (as far as I've seen) properly marked as cpython-only, but when some wacko decides the _testcapi module shouldn't, for example, be shipped to a million machines[*] that are never going to use it, it would be nice to still run the tests that can be run without _testcapi. Any objections to fixing the tests to use test.support.import_module() for _testcapi and a 'needs_testcapi' skipping decorator? To elaborate, we are also not shipping a couple of other modules (like distutils), but it's not unreasonable to expect those to exist (we modify the testsuite for that in our own builds only, instead, as well as making all our code deal with it.) The _testcapi module, however, is internal *and* meant for tests only, and used in quite a few tests (sometimes only in a single testfunction.) [*] 'a million machines' is not the actual number -- I don't know the actual number (but I'm sure it's bigger than that), I'm just tossing out some large number. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] performance of generator termination (was: Re: problem with recursive "yield from" delegation)
Antoine Pitrou, 08.03.2012 21:36:
> On Thu, 8 Mar 2012 14:36:06 -0600
> Benjamin Peterson wrote:
>> 2012/3/8 Stefan Behnel:
>>> Would that be acceptable for CPython as well or would you prefer full
>>> fledged normalisation?
>>
>> I think we have to normalize for correctness. Consider that it may be
>> some StopIteration subclass which set "value" on construction.
>
> Perhaps it would be time to drop the whole delayed normalization thing,
> provided the benchmarks don't exhibit a slowdown?
At least for Cython, always normalising the exception can make quite a
difference. For the nqueens benchmark, which uses short running generator
expressions (not affected by this particular change), a quick hack to
fetch, normalise and restore the StopIteration exception raised at the end
of the generator expression run reduces the performance by 10% for me. I'd
expect code similar to the group item iterator in itertools.groupby() to
suffer even worse for very small groups.
A while ago, I wouldn't have expected generator termination to have that an
impact, but when we dropped the single frame+traceback creation at the end
of the generator run in Cython, that boosted the performance of the
compiled nqueens benchmark by 70% and a compiled Python version of
itertools.groupby() ran twice as fast as before. These things can make an
impressively large difference.
http://thread.gmane.org/gmane.comp.python.cython.devel/12993/focus=13044
I'm using the following in Cython now. Note how complex the pre-3.3 case
is, I'm sure that makes it even more worth the special case in older
CPython versions (including 2.x).
"""
static int __Pyx_PyGen_FetchStopIterationValue(PyObject **pvalue) {
PyObject *et, *ev, *tb;
PyObject *value = NULL;
if (PyErr_ExceptionMatches(PyExc_StopIteration)) {
PyErr_Fetch(&et, &ev, &tb);
// most common case: plain StopIteration without argument
if (et == PyExc_StopIteration) {
if (!ev || !PyObject_IsInstance(ev, PyExc_StopIteration)) {
// PyErr_SetObject() puts the value directly into ev
if (!ev) {
Py_INCREF(Py_None);
ev = Py_None;
}
Py_XDECREF(tb);
Py_DECREF(et);
*pvalue = ev;
return 0;
}
}
// otherwise: normalise and check what that gives us
PyErr_NormalizeException(&et, &ev, &tb);
if (PyObject_IsInstance(ev, PyExc_StopIteration)) {
Py_XDECREF(tb);
Py_DECREF(et);
#if PY_VERSION_HEX >= 0x030300A0
value = ((PyStopIterationObject *)ev)->value;
Py_INCREF(value);
Py_DECREF(ev);
#else
PyObject* args = PyObject_GetAttrString(ev, "args");
Py_DECREF(ev);
if (args) {
value = PyObject_GetItem(args, 0);
Py_DECREF(args);
}
if (!value)
PyErr_Clear();
#endif
} else {
// looks like normalisation failed - raise the new exception
PyErr_Restore(et, ev, tb);
return -1;
}
} else if (PyErr_Occurred()) {
return -1;
}
if (value == NULL) {
Py_INCREF(Py_None);
value = Py_None;
}
*pvalue = value;
return 0;
}
"""
Stefan
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
I do not believe the change set below is valid.
As I read it, the new test verifies that one particular type of Nasty
key will provoke a RuntimeError -- but that particular type already
did so, by hitting the recursion limit. (It doesn't even really
mutate the dict.)
Meanwhile, the patch throws out tests for several different types of
mutations that have caused problems -- even segfaults -- in the past,
even after the dict implementation code was already "fixed".
Changing these tests to "assertRaises" would be fine, but they should
all be kept; if nothing else, they test whether you've caught all
mutation avenues.
-jJ
On Mon, Mar 5, 2012 at 7:13 PM, victor.stinner
wrote:
> http://hg.python.org/cpython/rev/934aaf2191d0
> changeset: 75445:934aaf2191d0
> user: Victor Stinner
> date: Tue Mar 06 01:03:13 2012 +0100
> summary:
> Close #14205: dict lookup raises a RuntimeError if the dict is modified
> during
> a lookup.
>
> "if you want to make a sandbox on top of CPython, you have to fix segfaults"
> so let's fix segfaults!
>
> files:
> Lib/test/crashers/nasty_eq_vs_dict.py | 47 --
> Lib/test/test_dict.py | 22 +-
> Lib/test/test_mutants.py | 291 --
> Misc/NEWS | 5 +-
> Objects/dictobject.c | 18 +-
> 5 files changed, 31 insertions(+), 352 deletions(-)
>
>
> diff --git a/Lib/test/crashers/nasty_eq_vs_dict.py
> b/Lib/test/crashers/nasty_eq_vs_dict.py
> deleted file mode 100644
> --- a/Lib/test/crashers/nasty_eq_vs_dict.py
> +++ /dev/null
> @@ -1,47 +0,0 @@
> -# from http://mail.python.org/pipermail/python-dev/2001-June/015239.html
> -
> -# if you keep changing a dictionary while looking up a key, you can
> -# provoke an infinite recursion in C
> -
> -# At the time neither Tim nor Michael could be bothered to think of a
> -# way to fix it.
> -
> -class Yuck:
> - def __init__(self):
> - self.i = 0
> -
> - def make_dangerous(self):
> - self.i = 1
> -
> - def __hash__(self):
> - # direct to slot 4 in table of size 8; slot 12 when size 16
> - return 4 + 8
> -
> - def __eq__(self, other):
> - if self.i == 0:
> - # leave dict alone
> - pass
> - elif self.i == 1:
> - # fiddle to 16 slots
> - self.__fill_dict(6)
> - self.i = 2
> - else:
> - # fiddle to 8 slots
> - self.__fill_dict(4)
> - self.i = 1
> -
> - return 1
> -
> - def __fill_dict(self, n):
> - self.i = 0
> - dict.clear()
> - for i in range(n):
> - dict[i] = i
> - dict[self] = "OK!"
> -
> -y = Yuck()
> -dict = {y: "OK!"}
> -
> -z = Yuck()
> -y.make_dangerous()
> -print(dict[z])
> diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py
> --- a/Lib/test/test_dict.py
> +++ b/Lib/test/test_dict.py
> @@ -379,7 +379,7 @@
> x.fail = True
> self.assertRaises(Exc, d.pop, x)
>
> - def test_mutatingiteration(self):
> + def test_mutating_iteration(self):
> # changing dict size during iteration
> d = {}
> d[1] = 1
> @@ -387,6 +387,26 @@
> for i in d:
> d[i+1] = 1
>
> + def test_mutating_lookup(self):
> + # changing dict during a lookup
> + class NastyKey:
> + mutate_dict = None
> +
> + def __hash__(self):
> + # hash collision!
> + return 1
> +
> + def __eq__(self, other):
> + if self.mutate_dict:
> + self.mutate_dict[self] = 1
> + return self == other
> +
> + d = {}
> + d[NastyKey()] = 0
> + NastyKey.mutate_dict = d
> + with self.assertRaises(RuntimeError):
> + d[NastyKey()] = None
> +
> def test_repr(self):
> d = {}
> self.assertEqual(repr(d), '{}')
> diff --git a/Lib/test/test_mutants.py b/Lib/test/test_mutants.py
> deleted file mode 100644
> --- a/Lib/test/test_mutants.py
> +++ /dev/null
> @@ -1,291 +0,0 @@
> -from test.support import verbose, TESTFN
> -import random
> -import os
> -
> -# From SF bug #422121: Insecurities in dict comparison.
> -
> -# Safety of code doing comparisons has been an historical Python weak spot.
> -# The problem is that comparison of structures written in C *naturally*
> -# wants to hold on to things like the size of the container, or "the
> -# biggest" containee so far, across a traversal of the container; but
> -# code to do containee comparisons can call back into Python and mutate
> -# the container in arbitrary ways while the C loop is in midstream. If the
> -# C code isn't extremely paranoid about digging things out of memory on
> -# each trip, and artificially boosting refcounts for the duration, anything
> -# from infinite loops to OS crashes can result (yes, I use Windows ).
> -#
> -# The other problem is that code designed to provoke a weakness is usually
Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
On 09/03/2012 22:32, Jim Jewett wrote: I do not believe the change set below is valid. As I read it, the new test verifies that one particular type of Nasty key will provoke a RuntimeError -- but that particular type already did so, by hitting the recursion limit. (It doesn't even really mutate the dict.) Oh yes, thanks for the report. I fixed that test. Meanwhile, the patch throws out tests for several different types of mutations that have caused problems -- even segfaults -- in the past, even after the dict implementation code was already "fixed". Changing these tests to "assertRaises" would be fine, but they should all be kept; if nothing else, they test whether you've caught all mutation avenues. I ran all these tests, none is still crashing. I don't think that it is interesting to keep them. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
Actually, I too noticed that you've dropped few crasher tests. I think we need to keep them, to make sure that future development will not introduce the same vulnerabilities. That's a common practice with unit-testing. On 2012-03-09, at 5:27 PM, Victor Stinner wrote: > On 09/03/2012 22:32, Jim Jewett wrote: >> I do not believe the change set below is valid. >> >> As I read it, the new test verifies that one particular type of Nasty >> key will provoke a RuntimeError -- but that particular type already >> did so, by hitting the recursion limit. (It doesn't even really >> mutate the dict.) > > Oh yes, thanks for the report. I fixed that test. > >> Meanwhile, the patch throws out tests for several different types of >> mutations that have caused problems -- even segfaults -- in the past, >> even after the dict implementation code was already "fixed". >> >> Changing these tests to "assertRaises" would be fine, but they should >> all be kept; if nothing else, they test whether you've caught all >> mutation avenues. > > I ran all these tests, none is still crashing. I don't think that it is > interesting to keep them. > > Victor > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
Victor Stinner wrote: On 09/03/2012 22:32, Jim Jewett wrote: I do not believe the change set below is valid. As I read it, the new test verifies that one particular type of Nasty key will provoke a RuntimeError -- but that particular type already did so, by hitting the recursion limit. (It doesn't even really mutate the dict.) Oh yes, thanks for the report. I fixed that test. Meanwhile, the patch throws out tests for several different types of mutations that have caused problems -- even segfaults -- in the past, even after the dict implementation code was already "fixed". Changing these tests to "assertRaises" would be fine, but they should all be kept; if nothing else, they test whether you've caught all mutation avenues. I ran all these tests, none is still crashing. I don't think that it is interesting to keep them. Aren't these regression tests? To be kept to make sure they don't fail in the future? ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testsuite dependency on _testcapi
On Fri, Mar 9, 2012 at 14:44, Thomas Wouters wrote: > > While testing Python 2.7 internally (at Google) I noticed that (now that > ImportErrors aren't automatically test skips) lots of tests fail if you > don't have the _testcapi module. These tests are (as far as I've seen) > properly marked as cpython-only, but when some wacko decides the _testcapi > module shouldn't, for example, be shipped to a million machines[*] that are > never going to use it, it would be nice to still run the tests that can be > run without _testcapi. Any objections to fixing the tests to use > test.support.import_module() for _testcapi and a 'needs_testcapi' skipping > decorator? > I have no issue with the test.support.import_module() use, although does it really require a full-on decorator? Is there a way to make it more generic to simply take a module name and if the import raises ImportError the test is skipped? -Brett > > To elaborate, we are also not shipping a couple of other modules (like > distutils), but it's not unreasonable to expect those to exist (we modify > the testsuite for that in our own builds only, instead, as well as making > all our code deal with it.) The _testcapi module, however, is internal > *and* meant for tests only, and used in quite a few tests (sometimes only > in a single testfunction.) > > [*] 'a million machines' is not the actual number -- I don't know the > actual number (but I'm sure it's bigger than that), I'm just tossing out > some large number. > -- > Thomas Wouters > > Hi! I'm a .signature virus! copy me into your .signature file to help me > spread! > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testsuite dependency on _testcapi
2012/3/9 Thomas Wouters : > > While testing Python 2.7 internally (at Google) I noticed that (now that > ImportErrors aren't automatically test skips) lots of tests fail if you > don't have the _testcapi module. These tests are (as far as I've seen) > properly marked as cpython-only, but when some wacko decides the _testcapi > module shouldn't, for example, be shipped to a million machines[*] that are > never going to use it, it would be nice to still run the tests that can be > run without _testcapi. Any objections to fixing the tests to use > test.support.import_module() for _testcapi and a 'needs_testcapi' skipping > decorator? Sounds fine to me. Post a patch. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
On Fri, Mar 09, 2012 at 05:38:05PM -0500, Yury Selivanov wrote: > Actually, I too noticed that you've dropped few crasher tests. I think > we need to keep them, to make sure that future development will not > introduce the same vulnerabilities. That's a common practice with > unit-testing. The term for this is "regression testing" -- when you fix a bug, you keep a test for that bug forever, to ensure that you never have a regression that re-introduces the bug. -- Steven ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testsuite dependency on _testcapi
On 09/03/2012 20:44, Thomas Wouters wrote: (...) it would be nice to still run the tests that can be run without _testcapi. Any objections to fixing the tests to use test.support.import_module() for _testcapi and a 'needs_testcapi' skipping decorator? test.support.import_module() looks fine for such issue. I'm guilty of adding new "from _testcapi import ..." in some tests (test_unicode at least). You add a test.support.import_module() in such test. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
