Bugs item #1529297, was opened at 2006-07-26 15:02
Message generated for change (Settings changed) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1529297&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: David MacQuigg (macquigg)
Assigned to: Nobody/Anonymous (nobody)
Summary: unrelated variable messing up doctests
Initial Comment:
Version 2.5b2 with IDLE 1.2b2 running on Windows XP
Home, fully updated.
Unpack the attached zip file.
Open program.py, test_suite.py, in separate IDLE
windows.
Run test_suite.py.
8 passed and 0 failed.
Now add a variable at the top of program.py, any name
will do.
xyz = 2
Re-run test_suite.py.
7 passed and 1 failed !!
I haven't been able to isolate the problem, but it is
repeatable. Taking out blocks of unrelated statements
changes the behavior. Feels like a memory allocation
error, or something strange.
----------------------------------------------------------------------
>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-07-27 21:37
Message:
Logged In: YES
user_id=33168
Closing again since the bug has been fixed. If you want a
feature request, you can open a new one.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-27 20:34
Message:
Logged In: YES
user_id=31435
Renaming the tests isn't a workaround in the absence of
sorting: nothing is guaranteed about __dict__ traversal
order, because nothing is defined about string hash codes.
You could suggest adding a way to force order as a new
feature request, but I won't look at it. It should simply
never be necessary for /reasonable/ tests. And, yes, that
means I think it's unreasonable to write tests that rely on
side effects across distinct tests occurring in a specific
order :-)
At this point I suggest you stop writing doctests in
function docstrings and use the newer doctest /file/
facilities instead. Then you can have a file full of tests
run in exactly the order you want -- although it's still
dubious practice to rely on side effects across distinct
sections of tests.
If you have to force order across function docstring tests,
you'll have to build your doctest driver out of lower-level
doctest components. That means, e.g., creating your own
instance of DocTestFinder, forcing the test order you want
in the list of test objects it creates, and feeding them
into a DocTestRunner instance.
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 20:09
Message:
Logged In: YES
user_id=676422
I've got a workaround. I changed the names of my test
functions to _1connect, _2hello, _3envfrom, _4envrcpt,
_5addheader, and _6eom.
Could we set the order explicitly, maybe
doctest.order = [connect, hello, envfrom, envrcpt, ... ]
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-27 18:31
Message:
Logged In: YES
user_id=31435
> would it be possible to run the tests in the order
> they are written, rather than in alphabetical order?
Sorry, that's impossible. doctest never sees the source
code, it crawls through module and class __dict__s to find
objects with docstrings, and the order things appear in
those is undefined.
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 18:03
Message:
Logged In: YES
user_id=676422
I hope this is not too much to ask, but would it be
possible to run the tests in the order they are written,
rather than in alphabetical order? This isn't a big deal,
as long as the order is stable, but it would abide by the
rule of least astonishment. :>)
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-27 17:53
Message:
Logged In: YES
user_id=31435
Glad it solved your mystery! Note that 2.4.4 and 2.5 /will/
sort the tests by name again (the code for that has already
been checked in) -- it was very intentional that it did that
before 2.4, and it was a bug that it stopped doing it in 2.4
through 2.4.3. OTOH, you're the only one so far who noticed :-)
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 17:46
Message:
Logged In: YES
user_id=676422
That fixed it!! Nice catch.
I've been aware for some time of the sensitivity to order
in the doctests, but it has never been a problem until now,
and I've been writing programs with doctests since 2.3.
When I add a test, and it breaks a later test, it has
always been easy to nail down the piece that changed.
This "behind the scenes" change of test order seems like
the kind of problem that rarely occurs, but can cause
serious problems when it does.
Thanks again for an excellent followup.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-27 17:01
Message:
Logged In: YES
user_id=31435
DocTestFinder has been changed to sort test names again, in
revs 50882 (on the trunk, for 2.5) and 50886 (on the 2.4
branch, for 2.4.4).
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-27 16:21
Message:
Logged In: YES
user_id=31435
I think I know what's happening here. doctest was largely
rewritten for Python 2.4, and it lost a subtle intended
feature: it used to (<= 2.3) sort tests by name before
running them, but in 2.4 started running them in whatever
order they happen to appear in the hash-code-based module
__dict__.
Changing the namespace in program.py in seemingly random
ways matters because test_suite.py starts with
from program import *
and so everything in program.py's module namespace
"pollutes" test_suite.py's namespace too. That can change
the order tests run in. And as I noted before, whether or
not the doctest for _envrcpt() passes or fails depends on
whether or not the doctests for _hello() run before them,
due to side effects on the shared `ci` instance. Look at
your output carefully, and I expect you'll find that the
order of tests /does/ change depending on crap variable
names you stick in (or remove from) program.py.
If you're with me so far, try changing the end of
doctest.py's DocTestFinder.find() from
return tests
to
tests.sort()
return tests
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 14:48
Message:
Logged In: YES
user_id=676422
Some observations:
If I give the test variable a private name, the problem
goes away.
. _xyz = 2
If I assign the variable to an object already in memory,
the problem doesn't go away.
. xyz = LocalADDRs
It doesn't matter that the test variable is never used. The
problem persists, even if I add a second statement:
. xyz = 2
. if xyz == 2: pass
It doesn't matter where in the module the statement is
placed, as long as it is module level, not inside a class
or function.
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 14:48
Message:
Logged In: YES
user_id=676422
Some observations:
If I give the test variable a private name, the problem
goes away.
. _xyz = 2
If I assign the variable to an object already in memory,
the problem doesn't go away.
. xyz = LocalADDRs
It doesn't matter that the test variable is never used. The
problem persists, even if I add a second statement:
. xyz = 2
. if xyz == 2: pass
It doesn't matter where in the module the statement is
placed, as long as it is module level, not inside a class
or function.
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 10:32
Message:
Logged In: YES
user_id=676422
The failure is not the specific output of doctest, but the
fact that it *changes* when I add an unrelated variable
xyz=2 in the module program.py. I am not able to reproduce
this failure with Tim's stripped down test_suite.py.
I agree the problem is not in IDLE. I can do the same from
a bash command line with
$ python2.4 test_suite.py
but not with
$ python2.3 test_suite.py
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-07-27 09:14
Message:
Logged In: YES
user_id=31435
I see exactly the same failure akuchling reported, on
Windows using current trunk, running from a command shell
(not IDLE -- didn't try IDLE). The attached test_suite.py
throws away everything not needed to see the same failure;
in particular, it gets rid of doctest, the logging module,
and most of the functions in test_suite.py.
Reduced to this, I don't see a reason to suspect a Python
bug. ci.IDrating is initialized to 100, and the
_envrcpt('[EMAIL PROTECTED]')
line produces (the expected)
CONTINUE [['[EMAIL PROTECTED]', 50, 2, 75, 101, '']]
if you /leave ci.IDrating alone/. However, if you do
ci.hello('zz_IP_192.168.0.8_mailout7.simple.tld')
first, that has a (apparently deliberate) side effect of
changing ci.IDrating to None, and then this path is /not/ taken:
# . . Whitelist pass
if self.IDrating > whitelvl:
rcptinfo[4] = 101 # ham > 100% to ignore later
spamscore ###2e
self.R.append(rcptinfo)
The 101 vanishes from the expected output then, and I get
CONTINUE [['[EMAIL PROTECTED]', 50, 2, 75, 50, '']]
instead.
That exactly reproduces the only failure Andrew & I have
seen, and appears to be what the code is telling it to do
(via side effects on the shared `ci` instance).
----------------------------------------------------------------------
Comment By: David MacQuigg (macquigg)
Date: 2006-07-27 08:22
Message:
Logged In: YES
user_id=676422
On my Redhat Linux machine, I see the failure in 2.4.1, but
not in 2.3.4.
Trying to isolate the problem, I've commented out some
blocks of unrelated code, but I can't get very far with
this, because many of the deletions make the problem go
away. It's as if the problem depends on the exact position
in memory of the test variable.
I did manage to delete the DNS queries, so now you can run
the tests without intermittent failures due to DNS timeouts.
I'll keep the latest at http://www.open-
mail.org/test/test_0607nn.tgz
Thanks for your efforts.
----------------------------------------------------------------------
Comment By: A.M. Kuchling (akuchling)
Date: 2006-07-27 05:15
Message:
Logged In: YES
user_id=11375
On a Linux machine, both 2.4.3 and SVN HEAD report 7 tests
passed, one failed.
For both versions, the failure is:
Failed example:
_envrcpt('[EMAIL PROTECTED]')
Expected:
CONTINUE [['[EMAIL PROTECTED]', 50, 2, 75, 101, '']]
Got:
CONTINUE [['[EMAIL PROTECTED]', 50, 2, 75, 50, '']]
----------------------------------------------------------------------
Comment By: Neal Norwitz (nnorwitz)
Date: 2006-07-26 20:59
Message:
Logged In: YES
user_id=33168
Can you try with SVN head? There was an unitialized read
that the doctests could trigger. It has been fixed.
Perhaps you are running into that?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1529297&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
