Re: [Python-Dev] 3.1.2

[Stefan Behnel]

Benjamin Peterson, 13.02.2010 03:52:
> It's about time for another 3.1 bug fix release. I propose this schedule:
> 
> March 6: Release Candidate (same day as 2.7a4)
> March 20:  3.1.2 Final release

Does a crash like #7173 qualify as a blocker for 3.1.2?

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.1.2

[Martin v. Löwis]

Stefan Behnel wrote:
> Benjamin Peterson, 13.02.2010 03:52:
>> It's about time for another 3.1 bug fix release. I propose this schedule:
>>
>> March 6: Release Candidate (same day as 2.7a4)
>> March 20:  3.1.2 Final release
> 
> Does a crash like #7173 qualify as a blocker for 3.1.2?

I'm not the release manager, but my feeling is that, because there is no
proposed resolution of the issue, it can't possibly be a blocker. Only
if a patch is available, waiting for application of that patch may block
the release. Waiting for a patch may cause indefinite delay, which would
be bad.

Of course, for releases managed by Barry (i.e. 2.6), Barry said that you
can declare anything a blocker - whether it then will block the release
is a different matter (and one that Barry then decides on a case-by-case
basis).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Google Groups Mirror

[anatoly techtonik]

On Mon, Feb 15, 2010 at 1:31 AM, Barry Warsaw  wrote:
>
>> What is the point in maintaining archive listed in
>> http://mail.python.org/mailman/listinfo/python-dev if it is not
>> searchable?
>> Recently I needed to find old thread about adding tags to roundup but 
>> couldn't.
>>
>> GMane archive doesn't search -
>> http://news.gmane.org/navbar.php?group=gmane.comp.python.devel&query=roundup+tags
>> and Google Group archive is private http://groups.google.com/group/python-dev
>>
>> How about opening an official read-only Google Group mirror? Then it
>> will be possible to subscribe to notifications to just interesting
>> threads.
>
> Try mail-archive.com, it's searchable.
>
> http://www.mail-archive.com/python-dev@python.org/

Thanks. Still an official search form will be welcomed as well as
per-thread subscriptions, and not only for python-dev.

Crossposted to postmas...@python.org

-- 
anatoly t.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Release timer for Core Development page

[anatoly techtonik]

I've got another idea of having a release timer on
http://python.org/dev/ page together with link to generated release
calendar.
It will help to automatically monitor deadlines for feature fixes in
alpha releases without manually monitoring this mailing list.

There is already a navigation box on the right side where this
information fits like a glove.
Does anybody else find this feature useful for Python development?

-- 
anatoly t.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 13/02/2010 11:00, Robert Collins wrote:

On Sat, 2010-02-13 at 10:42 +, Antoine Pitrou wrote:
   

Robert Collins  robertcollins.net>  writes:
 

I'm not personally very keen on inspecting everything in self.__dict__,
I suspect it would tickle bugs in other unittest extensions. However I'm
not really /against/ it - I don't think it will result in bad test
behaviour or isolation issues. So if users would like it, lets do it.
   

Why not take all resource_XXX attributes?
 

Sure, though if we're just introspecting I don't see much risk
difference between resource_XXX and all XXX. As I say above I'm happy to
do it if folk think it will be nice.
   


We could introspect all class attributes and find all the resources. We 
should use dir(...) rather than looking in the class __dict__ so that 
resources can be inherited.


However, it sounds like Guido isn't a fan of Test Resources *instead* of 
setUpClass / module because it doesn't offer a simple solution for the 
setUpModule use case and the API still needs to mature.


All the best,

Michael

   

By the way, how does a given test access the allocated resource? Say, the DB
connection. Does it become an attribute of the test case instance?
 

yes. Given

class Foo(TestCase):
 resources = {'thing', MyResourceManager()}
 def test_foo(self):
 self.thing

self.thing will access the resource returned by MyResourceManager.make()

-Rob
   



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
   



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your 
employer, to release me from all obligations and waivers arising from any and all 
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, 
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS 
AGREEMENTS") that I have entered into with your employer, its partners, licensors, 
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. 
You further represent that you have the authority to release me from any BOGUS AGREEMENTS 
on behalf of your employer.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 13/02/2010 04:01, Glyph Lefkowitz wrote:
[snipping some good points...]

Regarding the objection that setUp/tearDown for classes would run into
issues with subclassing, I propose to let the standard semantics of
subclasses do their job. Thus a subclass that overrides setUpClass or
tearDownClass is responsible for calling the base class's setUpClass
and tearDownClass (and the TestCase base class should provide empty
versions of both). The testrunner should only call setUpClass and
tearDownClass for classes that have at least one test that is
selected.

Yes, this would mean that if a base class has a test method and a
setUpClass (and tearDownClass) method and a subclass also has a test
method and overrides setUpClass (and/or tearDown), the base class's
setUpClass and tearDown may be called twice. What's the big deal? If
setUpClass and tearDownClass are written properly they should support
this.
 

Just to be clear: by "written properly" you mean, written as classmethods, 
storing their data only on 'cls', right?

   


Heh, yes (answered several times in this thread already I think...).


If this behavior is undesired in a particular case, maybe what
was really meant were module-level setUp and tearDown, or the class
structure should be rearranged.
 

There's also a bit of an open question here for me: if subclassing is allowed, 
and module-level setup and teardown are allowed, then what if I define a test 
class with test methods in module 'a', as well as module setup and teardown, 
then subclass it in 'b' which *doesn't* have setup and teardown... is the 
subclass in 'b' always assumed to depend on the module-level setup in 'a'?  Is 
there a way that it could be made not to if it weren't necessary? What if it 
stubs out all of its test methods?  In the case of classes you've got the 'cls' 
variable to describe the dependency and the shared state, but in the case of 
modules, inheritance doesn't create an additional module object to hold on to.

   
This is also an interesting point. The 'naive' implementation, which I 
think I prefer, only runs the setUpModule of modules actually containing 
tests. Similarly setUpClass is only called on classes with actual tests, 
although they may call up to base class implementations.


This has a couple of consequences, particularly for setUpModule. It 
makes the basic rule:


* only use setUpModule for modules actually containing tests
* don't mix concrete TestCases (with tests) in the same module (using 
setUpModule) as base classes for other tests


The use case (that I can think of) that isn't supported (and of course 
all this needs to be documented):


* Having setUpModule / tearDownModule in modules that define base 
classes and expecting them to be called around all tests that inherit 
from those base classes


Having this in place makes the implementation simpler. If we explicitly 
document that this rule may change and so users shouldn't rely on 
setUpModule not being called for modules containing base classes, then 
we are free to rethink it later. Not having this restriction at all is 
possible, it just requires more introspection at TestSuite creation / 
ordering time.


Note that setUpClass on a base class maybe called several times if 
several base classes inherit and all call up to the base class 
implementation. As it will be a class method the cls argument will be 
different for each call.


Another question. If we are implementing TestCase.setUpClass as an 
additional test then should it be reported *even* if it is only the 
default (empty) implementation that is used?


The reason to have setUpClass implemented as a test is so that you can 
report the failure *before* you run all the tests. Lots of unit test 
users want a consistent number of tests every run - so we shouldn't 
insert an extra test only on fail. The other alternative is to report a 
setUpClass failure as part of the first test that depends on it - this 
makes the implementation more, complex (having a pseudo-test represent 
setUpClass / tearDownClass is convenient for ordering and isolating the 
'magic' in one place - the rest of the test running infrastructure 
doesn't need to know about setUpClass or Module).


If we do add a default setUpClass test for all TestCases it means extra 
noise for test runs that don't use the new feature. Saying no it 
shouldn't be shown means that we have to introspect test classes to see 
if they inherit setUpClass from TestCase or from some intermediate base 
class. Not hard just an extra complexity. One solution would be for 
TestCase *not* to have default implementations, but it is probably nicer 
for them to appear in the API.


I guess my preferred approach is to have a default implementation, but 
not to create pseudo-tests for them if they aren't used.



All the best,

Michael


testresources very neatly sidesteps this problem by just providing an API to say 
"this test case depends on that test resource", without relying on the gro

[Python-Dev] pysandsox project

[Victor Stinner]

Hi,

I'm working on a new sandbox project. The goal is to create an empty namespace 
and write strict rules for the interaction with the existing namespace (full 
featured Python namespace).

By default, you cannot read a file, use print, import a module or exit Python. 
But you can enable some functions by using config "features". 
Example: "regex" feature allows you to import the re module which will 
contain a subset of the real re module, just enough to match a regex.

To protect the sandbox namespace, some attributes are "hidden": function 
closure and globals, frame locals and type subclasses. __builtins__ is also 
replaced by a read-only dictionary. Objects are not directly injected in the 
sandbox namespace: a proxy is used to get a read-only view of the object.

pysandbox is based on safelite.py, project written by tav one year ago (search 
tav in python-dev archive, February 2009). I tested RestrictedPython, but the 
approach is different (rewrite bytecode) and the project is not maintained 
since 3 or 4 years (only minor updates on the documentation or the copyright 
header). pysandbox is different than RestrictedPython because it blocks 
everything by default and has simply config options to enable a set of 
features. pysandbox is different than safelite.py because it contains unit 
tests (ensure that blocked features are really blocked).

Only attributes/functions allowing to escape the sandbox are blocked. Eg. 
frames are still accessibles, only the frame locals are blocked. This 
blacklist policy is broken by design, but it's a nice way to quickly get a 
working sandbox without having to modify CPython too much.

pysandbox status is closer to a proof-of-concept than a beta version, there 
are open issues (see above). Please test it and try to break it!

--

To try pysandbox, download the last version using git clone or a tarball at:

http://github.com/haypo/pysandbox/

You don't need to install it, use "python interpreter.py" or "python 
execfile.py yourscript.py". Use --help to get more options.

I tested pysandbox on Linux with Python 2.5, 2.6 and 2.7. I guess that it 
should work on Python 3.0 with minor changes.

--

The current syntax is:

   config = SandboxConfig(...)
   with Sandbox(config):
   ... execute untrusted code here ...

This syntax has a problem: local frame variables are not protected by a proxy, 
nor removed from the sandbox namespace. I tried to remove the frame locals, 
but Python uses STORE_FAST/LOAD_FAST bytecodes in a function, and this fast 
cache is not accessible in Python. Clear this cache may introduce unexpected 
behaviours.

pysandbox modify some structure attributes (frame.f_builtins and 
frame.f_tstate.interp.builtins) directly in memory using some ctypes tricks. 
I used that to avoid patching CPython and to get faster a working 
proof-of-concept. Set USE_CPYTHON_HACK to False (in sandbox/__init__.py) to 
disable these hacks, but they are needed to protect __builtins__ (see related 
tests).

--

By default, pysandbox doesn't use CPython restricted mode, because this mode 
is too restrictive (it's not possible to read a file or import a module). But 
pysandbox can use it with SandboxConfig(cpython_restricted=True).

--

See README file for more information and TODO file for a longer status.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 15/02/2010 17:05, Michael Foord wrote:

[snip]
This is also an interesting point. The 'naive' implementation, which I 
think I prefer, only runs the setUpModule of modules actually 
containing tests. Similarly setUpClass is only called on classes with 
actual tests, although they may call up to base class implementations.


This has a couple of consequences, particularly for setUpModule. It 
makes the basic rule:


* only use setUpModule for modules actually containing tests
* don't mix concrete TestCases (with tests) in the same module (using 
setUpModule) as base classes for other tests


The use case (that I can think of) that isn't supported (and of course 
all this needs to be documented):


* Having setUpModule / tearDownModule in modules that define base 
classes and expecting them to be called around all tests that inherit 
from those base classes


Having this in place makes the implementation simpler. If we 
explicitly document that this rule may change and so users shouldn't 
rely on setUpModule not being called for modules containing base 
classes, then we are free to rethink it later. Not having this 
restriction at all is possible, it just requires more introspection at 
TestSuite creation / ordering time.


Note that setUpClass on a base class maybe called several times if 
several base classes inherit and all call up to the base class 
implementation. As it will be a class method the cls argument will be 
different for each call.


Another question. If we are implementing TestCase.setUpClass as an 
additional test then should it be reported *even* if it is only the 
default (empty) implementation that is used?


The reason to have setUpClass implemented as a test is so that you can 
report the failure *before* you run all the tests. Lots of unit test 
users want a consistent number of tests every run - so we shouldn't 
insert an extra test only on fail. The other alternative is to report 
a setUpClass failure as part of the first test that depends on it - 
this makes the implementation more, complex (having a pseudo-test 
represent setUpClass / tearDownClass is convenient for ordering and 
isolating the 'magic' in one place - the rest of the test running 
infrastructure doesn't need to know about setUpClass or Module).


If we do add a default setUpClass test for all TestCases it means 
extra noise for test runs that don't use the new feature. Saying no it 
shouldn't be shown means that we have to introspect test classes to 
see if they inherit setUpClass from TestCase or from some intermediate 
base class. Not hard just an extra complexity. One solution would be 
for TestCase *not* to have default implementations, but it is probably 
nicer for them to appear in the API.


I guess my preferred approach is to have a default implementation, but 
not to create pseudo-tests for them if they aren't used.




One place to implement this is in the TestLoader (specifically in 
loadTestsFromModule and loadTestsFromTestCase) - especially as this is 
the place where test ordering is currently provided by unittest. The 
TestSuite may not need to change much. This isn't compatible with test 
frameworks that build custom suites without going through 
loadTestsFromModule - e.g. modules implementing load_tests that replace 
the standard test suite with a new suite with specific TestCases and 
still expect setUpModule / tearDownModule to be used [1]. Perhaps an API 
hook to make this easy?


TestCase would need a change in run(...) to support automatic 
SetupFailed test failing when setUpClass / module fails.


This *isn't* compatible with custom TestSuites that reorder tests. The 
alternative is for TestSuite.run(...) to change to support setUpClass / 
setUpModule by adding them at test run time by introspecting contained 
tests. That would make setUpClass (etc) incompatible with custom 
TestSuite implementations that override run. It would be compatible with 
TestSuites that do reordering but that don't overload run(...) - 
although the resulting test order may not be 'optimal' (the setUp and 
tearDown may end up separated farther in time than desired but at least 
they would work). Perhaps this approach is less likely to break? 
(Manually reordering tests or adding new tests to a TestSuite wouldn't 
break setUpClass as the TestSuite only adds them at run(...) time rather 
than placing them in the suite where the 'user' can move them around).


All the best,

Michael

[1] or even load_tests that just addTests to the standard suite but 
still need tearDownModule to be run *after* all the tests. Even with the 
implementation in loadTestsFrom* it seems like the TestSuite may need to 
know something about the order. Perhaps we could have 
TestLoader.orderTests - the loader is available to load_tests functions.





All the best,

Michael

testresources very neatly sidesteps this problem by just providing an 
API to say "this test case depends on that test resource", without 
relying on the grouping of tes

Re: [Python-Dev] 3.1.2

[Benjamin Peterson]

2010/2/15 "Martin v. Löwis" :
> Stefan Behnel wrote:
>> Benjamin Peterson, 13.02.2010 03:52:
>>> It's about time for another 3.1 bug fix release. I propose this schedule:
>>>
>>> March 6: Release Candidate (same day as 2.7a4)
>>> March 20:  3.1.2 Final release
>>
>> Does a crash like #7173 qualify as a blocker for 3.1.2?
>
> I'm not the release manager, but my feeling is that, because there is no
> proposed resolution of the issue, it can't possibly be a blocker. Only
> if a patch is available, waiting for application of that patch may block
> the release. Waiting for a patch may cause indefinite delay, which would
> be bad.

I agree with Martin here. I would be more inclined to make #7173 a
release blocker if it had a more specific test than "run cython and
maybe it'll crash".



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pysandsox project

[Hanno Schlichting]

On Mon, Feb 15, 2010 at 6:11 PM, Victor Stinner
 wrote:
> pysandbox is based on safelite.py, project written by tav one year ago (search
> tav in python-dev archive, February 2009). I tested RestrictedPython, but the
> approach is different (rewrite bytecode) and the project is not maintained
> since 3 or 4 years (only minor updates on the documentation or the copyright
> header).

Not that it matters much, but RestrictedPython [1] is pretty well
maintained for a seasoned library.

That doesn't mean it's a particular good solution to the problem, just
that it works for the use-case it was written for and is still in
widespread use as part of Zope / Plone.

Hanno

[1] http://pypi.python.org/pypi/RestrictedPython
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.1.2

[Amaury Forgeot d'Arc]

Hi,

2010/2/15 Benjamin Peterson 

> 2010/2/15 "Martin v. Löwis" :
> > Stefan Behnel wrote:
> >> Benjamin Peterson, 13.02.2010 03:52:
> >>> It's about time for another 3.1 bug fix release. I propose this
> schedule:
> >>>
> >>> March 6: Release Candidate (same day as 2.7a4)
> >>> March 20:  3.1.2 Final release
> >>
> >> Does a crash like #7173 qualify as a blocker for 3.1.2?
> >
> > I'm not the release manager, but my feeling is that, because there is no
> > proposed resolution of the issue, it can't possibly be a blocker. Only
> > if a patch is available, waiting for application of that patch may block
> > the release. Waiting for a patch may cause indefinite delay, which would
> > be bad.
>
> I agree with Martin here. I would be more inclined to make #7173 a
> release blocker if it had a more specific test than "run cython and
> maybe it'll crash".
>

I just updated #7173 with a short crasher.

In short, I think that next() in an exception handler messes with
the exception state.
This doesn't play well with the cyclic garbage collector which can call
tp_clear()
on a resurrected object, if the reference is *moved* out of the cycle by
some tp_dealloc.

-- 
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Glyph Lefkowitz]

On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:

> On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
>  wrote:
>> On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
>> 
>> For what it's worth, I am a big fan of abusing test frameworks in generally, 
>> and pyunit specifically, to perform every possible kind of testing.  In 
>> fact, I find setUpClass more hostile to *other* kinds of testing, because 
>> this convenience for simple integration tests makes more involved, 
>> performance-intensive integration tests harder to write and manage.
> 
> That sounds odd, as if the presence of this convenience would prohibit
> you from also implement other features.

Well, that is the main point I'm trying to make.  There are ways to implement 
setUpClass that *do* make the implementation of other features effectively 
impossible, by breaking the integration mechanisms between tests and framework, 
and between multiple testing frameworks.

And I am pretty sure this is not just my over-reaction; Michael still appears 
to be wrestling with the problems I'm describing.  In a recent message he was 
talking about either breaking compatibility with TestSuite implementations that 
override run(), or test-reordering - both of which I consider important, core 
features of the unittest module.

>> I tried to write about this problem a while ago 
>>  - the current extensibility API 
>> (which is mostly just composing "run()") is sub-optimal in many ways, but 
>> it's important not to break it.
> 
> I expect that *eventually* something will come along that is so much
> better than unittest that, once matured, we'll want it in the stdlib.

I'm not sure what point you're trying to make here.  I was saying "it's not 
perfect, but we should be careful not to break it, because it's all we've got". 
 Are you saying that we shouldn't worry about unittest's composition API, 
because it's just a stopgap until something better comes along?

> (Or, alternatively, eventually stdlib inclusion won't be such a big
> deal any more since distros mix and match. But then inclusion in a
> distro would become every package developer's goal -- and then the
> circle would be round, since distros hardly move faster than Python
> releases...)
> 
> But in the mean time I believe evolving unittest is the right thing to
> do. Adding new methods is relatively easy. Adding whole new paradigms
> (like testresources) is a lot harder, eventually in the light of the
> latter's relative immaturity.

I disagree with your classification of the solutions.

First and foremost: setUpClass is not a "new method", it's a pile of new code 
to call that method, to deal with ordering that method, etc.  Code which has 
not yet been written or tested or tried in the real world. It is beyond simply 
immature, it's hypothetical.  We do have an implementation of this code in 
Twisted, but as I have said, it's an albatross we are struggling to divest 
ourselves of, not something we'd like to propose for inclusion in the standard 
library.  (Nose has this feature as well, but I doubt their implementation 
would be usable, since their idea of a 'test' isn't really TestCase based.)

testresources, by contrast, is a tested, existing package, which people are 
already using, using a long-standing integration mechanism that has been part 
of unittest since its first implementation.  Granted, I would not contest that 
it is "immature"; it is still fairly new, and doesn't have a huge number of 
uses, but it's odd to criticize it on grounds of maturity when it's so much 
*more* mature than the alternative.

While superficially the programming interface to testresources is slightly more 
unusual, this is only because programmers don't think to hard about what 
unittest actually does with your code, and testresources requires a little more 
familiarity with that.

>> And setUpClass does inevitably start to break those integration points down, 
>> because it implies certain things, like the fact that classes and modules 
>> are suites, or are otherwise grouped together in test ordering.

> I expect that is what the majority of unittest users already believe.

Yes, but they're wrong, and enforcing this misconception doesn't help anyone.  
There are all kinds of assumptions that most python developers have about how 
Python works which are vaguely incorrect abstractions over the actual behavior.

>> This makes it difficult to create custom suites, to do custom ordering, 
>> custom per-test behavior (like metrics collection before and after run(), or 
>> gc.collect() after each test, or looking for newly-opened-but-not-cleaned-up 
>> external resources like file descriptors after each tearDown).
> 
> True, the list never ends.
> 
>> Again: these are all concrete features that *users* of test frameworks want, 
>> not just idle architectural fantasy of us framework hackers.
> 
> I expect that most bleeding edge users will end up writing a custom
> f

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 15/02/2010 20:27, Glyph Lefkowitz wrote:


On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:


On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
mailto:gl...@twistedmatrix.com>> wrote:

On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:

For what it's worth, I am a big fan of abusing test frameworks in 
generally, and pyunit specifically, to perform every possible kind 
of testing.  In fact, I find setUpClass more hostile to *other* 
kinds of testing, because this convenience for simple integration 
tests makes more involved, performance-intensive integration tests 
harder to write and manage.


That sounds odd, as if the presence of this convenience would prohibit
you from also implement other features.


Well, that is the main point I'm trying to make.  There are ways to 
implement setUpClass that *do* make the implementation of other 
features effectively impossible, by breaking the integration 
mechanisms between tests and framework, and between multiple testing 
frameworks.


And I am pretty sure this is not just my over-reaction; Michael still 
appears to be wrestling with the problems I'm describing.

And I appreciate your input.
 In a recent message he was talking about either breaking 
compatibility with TestSuite implementations that override run(), or 
test-reordering - both of which I consider important, core features of 
the unittest module.


Well, by "breaking compatibility with custom TestSuite implementations 
that override run" I mean that is one possible place to put the 
functionality. Code that does override it will *not* stop working, it 
just won't support the new features.


If we chose this implementation strategy there would be no compatibility 
issues for existing tests / frameworks that don't use the new features. 
If tests do want to use the new features then the framework authors will 
need to ensure they are compatible with them. This seems like a 
reasonable trade-off to me. We can ensure that it is easy to write 
custom TestSuite objects that work with earlier versions of unittest but 
are also compatible with setUpClass in 2.7 (and document the recipe - 
although I expect it will just mean that TestSuite.run should call a 
single method if it exists).


Perhaps a better idea might be to also add startTest and stopTest 
methods to TestSuite so that frameworks can build in features like 
timing tests (etc) without having to override run itself. This is 
already possible in the TestResult of course, which is a more common 
extensibility point in *my* experience.


All the best,

Michael




I tried to write about this problem a while ago 
 - the current extensibility 
API (which is mostly just composing "run()") is sub-optimal in many 
ways, but it's important not to break it.


I expect that *eventually* something will come along that is so much
better than unittest that, once matured, we'll want it in the stdlib.


I'm not sure what point you're trying to make here.  I was saying 
"it's not perfect, but we should be careful not to break it, because 
it's all we've got".  Are you saying that we shouldn't worry about 
unittest's composition API, because it's just a stopgap until 
something better comes along?



(Or, alternatively, eventually stdlib inclusion won't be such a big
deal any more since distros mix and match. But then inclusion in a
distro would become every package developer's goal -- and then the
circle would be round, since distros hardly move faster than Python
releases...)

But in the mean time I believe evolving unittest is the right thing to
do. Adding new methods is relatively easy. Adding whole new paradigms
(like testresources) is a lot harder, eventually in the light of the
latter's relative immaturity.


I disagree with your classification of the solutions.

First and foremost: setUpClass is not a "new method", it's a pile of 
new code to call that method, to deal with ordering that method, etc. 
 Code which has not yet been written or tested or tried in the real 
world. It is beyond simply immature, it's hypothetical.  We do have an 
implementation of this code in Twisted, but as I have said, it's an 
albatross we are struggling to divest ourselves of, not something we'd 
like to propose for inclusion in the standard library.  (Nose has this 
feature as well, but I doubt their implementation would be usable, 
since their idea of a 'test' isn't really TestCase based.)


testresources, by contrast, is a tested, existing package, which 
people are already using, using a long-standing integration mechanism 
that has been part of unittest since its first implementation. 
 Granted, I would not contest that it is "immature"; it is still 
fairly new, and doesn't have a huge number of uses, but it's odd to 
criticize it on grounds of maturity when it's so much *more* mature 
than the alternative.


While superficially the programming interface to testresources is 
slightly more unusual, this is only because prog

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Glyph Lefkowitz]

On Feb 15, 2010, at 3:50 PM, Michael Foord wrote:

> On 15/02/2010 20:27, Glyph Lefkowitz wrote:
>> 
>> 
>> On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
>> 
>>> On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
>>>  wrote:
 I find setUpClass more hostile to *other* kinds of testing, because this 
 convenience for simple integration tests makes more involved, 
 performance-intensive integration tests harder to write and manage.
>>> 
>>> That sounds odd, as if the presence of this convenience would prohibit
>>> you from also implement other features.
>> 
>> And I am pretty sure this is not just my over-reaction; Michael still 
>> appears to be wrestling with the problems I'm describing.
> And I appreciate your input.

Thanks :).

>>  In a recent message he was talking about either breaking compatibility with 
>> TestSuite implementations that override run(), or test-reordering - both of 
>> which I consider important, core features of the unittest module.
> 
> Well, by "breaking compatibility with custom TestSuite implementations that 
> override run" I mean that is one possible place to put the functionality. 
> Code that does override it will *not* stop working, it just won't support the 
> new features.

Ah, I see.  This doesn't sound *too* bad, but I'd personally prefer it if the 
distinction were a bit more clearly drawn.  I'd like frameworks to be able to 
implement extension functionality without having to first stub out 
functionality.  In other words, if I want a test suite without setUpClass, I'd 
prefer to avoid having an abstraction inversion.

Practically speaking this could be implemented by having a very spare, basic 
TestSuite base class and ClassSuite/ModuleSuite subclasses which implement the 
setUpXXX functionality.

> If we chose this implementation strategy there would be no compatibility 
> issues for existing tests / frameworks that don't use the new features.

That's very good to hear.

> If tests do want to use the new features then the framework authors will need 
> to ensure they are compatible with them. This seems like a reasonable 
> trade-off to me. We can ensure that it is easy to write custom TestSuite 
> objects that work with earlier versions of unittest but are also compatible 
> with setUpClass in 2.7 (and document the recipe - although I expect it will 
> just mean that TestSuite.run should call a single method if it exists).

This is something that I hope Jonathan Lange or Robert Collins will chime in to 
comment on: expanding the protocol between suite and test is an area which is 
fraught with peril, but it seems like it's something that test framework 
authors always want to do.  (Personally, *I* really want to do it because I 
want to be able to run things asynchronously, so the semantics of 'run()' need 
to change pretty dramatically to support that...)  It might be good to 
eventually develop a general mechanism for this, rather than building up an 
ad-hoc list of test-feature compatibility recipes which involve a list of if 
hasattr(...): foo(); checks in every suite implementation.

> Perhaps a better idea might be to also add startTest and stopTest methods to 
> TestSuite so that frameworks can build in features like timing tests (etc) 
> without having to override run itself. This is already possible in the 
> TestResult of course, which is a more common extensibility point in *my* 
> experience.

I think timing and monitoring tests can mostly be done in the TestResult class; 
those were bad examples.  There's stuff like synthesizing arguments for test 
methods, or deciding to repeat a potentially flaky test method before reporting 
a failure, which are not possible to do from the result.  I'm not sure that 
startTest and stopTest hooks help with those features, the ones which really 
need suites; it would seem it mostly gives you a hook to do stuff that could 
already be done in TestResult anyway.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com