[issue16748] Make CPython test package discoverable

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue16748] Make CPython test package discoverable

Changes by Zachary Ware : -- dependencies: +Make full use of test discovery in test subpackages versions: +Python 3.5 -Python 3.3 ___ Python tracker ___ _

[issue16748] Make CPython test package discoverable

Changes by Zachary Ware : -- dependencies: +Clean up/refactor/make discoverable test_decimal, Fix test discovery for test_codecmaps*.py, Fix test discovery for test_concurrent_futures.py, Support ./python -m unittest in test_socket ___ Python tracke

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: So many things have been fixed since I last made a list of failing modules that it seemed like time to make a new one. First, failing modules that already have open issues with patches: - test_codecmaps* (issue18258) - test_concurrent_futures (issue16968) - tes

[issue16748] Make CPython test package discoverable

Terry J. Reedy added the comment: Chris: >>> import unittest as u; u.main("test.test_xxx") Ezio: I think that's the last nail in the coffin of test_main. Terry, do you agree? Belated answer: now that I use and understand the idiom, yes. Chris: The load_tests protocol (2.7, 3.2+) seems like the

[issue16748] Make CPython test package discoverable

Changes by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: Okay, well the title of this issue is "Make CPython test package discoverable," so as part of that we should be confirming that test discovery finds the same tests that test_main() runs (and if not, to understand why and/or make them the same). -- __

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: For the conversion from test_main() to unittest.main(), I could certainly see automation helping a lot; most cases are very simple. But really, that issue is somewhat tangential to this one and in my last message I was thinking from the perspective of just the

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: > Also, I'm of the opinion that we'll end up with a higher quality result doing > things by hand anyway. Automation doesn't (and shouldn't) preclude careful review and modifications by hand. My point was that it seems like it might be able to get you, say, 80

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: Chris: > There are lots of modules to change here. I wonder if some or most of > this couldn't be automated. Possibly, but I don't mind going through individually if Ezio (or others) don't mind committing individually. From what I've seen, the test suite is va

[issue16748] Make CPython test package discoverable

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file28690/test_overriding.txt ___ Python tracker ___ ___ Python-bugs-list mai

[issue16748] Make CPython test package discoverable

Serhiy Storchaka added the comment: Here is a dirty patch which hacks unittest to search possible test overriding. Just apply the patch and run regression tests. -- Added file: http://bugs.python.org/file28689/checkTestOverriding.diff ___ Python trac

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: As suggested in the previous comment, here is simple code to find candidates for test duplication (TestCase subclasses subclassing other TestCase classes): def find_dupes(mod): objects = [getattr(mod, name) for name in sorted(dir(mod))] classes = [obj f

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: There are lots of modules to change here. I wonder if some or most of this couldn't be automated. For example, is there any reason we couldn't write a script to check for the type of test duplication fixed documentation-wise in issue 16835? We could use suc

[issue16748] Make CPython test package discoverable

Ezio Melotti added the comment: > There are tests outside of Lib/test/ hierarchy. See #10572. -- ___ Python tracker ___ ___ Python-bu

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: > Also, it may be possible to add unittest discovery hooks to the stubs that > *are* in Lib/test. The load_tests protocol (2.7, 3.2+) seems like the right approach for this: http://docs.python.org/dev/library/unittest.html#load-tests-protocol --

[issue16748] Make CPython test package discoverable

Brett Cannon added the comment: It's totally doable to set up stubs in Lib/test to use test discovery on those tests which exist outside of that directory. test_importlib actually has some code for that left over from when it lived in Lib/importlib/test: http://hg.python.org/cpython/file/4617b

[issue16748] Make CPython test package discoverable

R. David Murray added the comment: Also, it may be possible to add unittest discovery hooks to the stubs that *are* in Lib/test. -- ___ Python tracker ___ __

[issue16748] Make CPython test package discoverable

R. David Murray added the comment: Yes, but not many, and not as many as there used to be. I'd like to see them all moved, but met resistance on that front. It may be that those just can't be run using unittest discovery, and perhaps that will eventually convince the maintainers to move them

[issue16748] Make CPython test package discoverable

Serhiy Storchaka added the comment: There are tests outside of Lib/test/ hierarchy. -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue16748] Make CPython test package discoverable

Ezio Melotti added the comment: I think that's the last nail in the coffin of test_main. Terry, do you agree? -- ___ Python tracker ___ __

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: One more. Since unittest imports strings, you can also do: >>> import unittest as u; u.main("test.test_xxx") -- ___ Python tracker ___ ___

[issue16748] Make CPython test package discoverable

Chris Jerdonek added the comment: > >>> import test.test_xxx as t; t.test_main() As long as the module has "import unittest", you could also do the following, which has just 5 more characters :) >>> import test.test_xxx as t; t.unittest.main(t) -- nosy: +chris.jerdonek __

[issue16748] Make CPython test package discoverable

Ezio Melotti added the comment: Without test_main you would have to do unittest.main(t, exit=False). I'm not sure there's an easier way. -- ___ Python tracker ___ _

[issue16748] Make CPython test package discoverable

R. David Murray added the comment: As we discussed in another issue, you just need to change your pattern to: >>> import unittest.main as runtest >>> import test.test_xxx as t >>> runtest(t) Which granted is more typing if you are running just one test, but not much more if you are running mor

[issue16748] Make CPython test package discoverable

Terry J. Reedy added the comment: test_main makes it trivial to import a test file and run the test. >>> import test/test_xxx as t; t.test_main() I do not know the implications of unittest.main(), but I would dislike losing the above ability. -- ___

[issue16748] Make CPython test package discoverable

Ezio Melotti added the comment: While we are at it, should we also move these tests to use unittest.main() instead of test_main() and similar? -- dependencies: +Fix test discovery for test_array.py ___ Python tracker

[issue16748] Make CPython test package discoverable

R. David Murray added the comment: Great list, thanks. The ones that fail to be run/skipped when run under discovery can probably be fixed by moving them to the more modern unittest 'skip' functions instead of depending on being run under regrtest and using the test.support resource functions

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: I've come up with a semi-comprehensive list of the tests that cause ugly failures with test discovery. Tests were run on 3.4 debug, on Windows 7 32bit without several of the 'external' projects built, using the command ``PCbuild\python_d.exe -m unittest discove

[issue16748] Make CPython test package discoverable

Ezio Melotti added the comment: Feel free to add me to the nosy. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue16748] Make CPython test package discoverable

Changes by Stefan Krah : -- nosy: -skrah ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue16748] Make CPython test package discoverable

Changes by Brett Cannon : -- dependencies: +Fix test discovery for test_genericpath.py ___ Python tracker ___ ___ Python-bugs-list mai

[issue16748] Make CPython test package discoverable

Brett Cannon added the comment: You can go ahead and start a new issue so it isn't forgotten about as this becomes a meta issue. And you can always add me to the nosy, just can't guarantee how fast I will do the reviews. =) -- ___ Python tracker <

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: Sounds good to me. Shall I move the genericpath fix to a new issue, or leave that one here and begin starting new issues with the next one tackled? Any volunteers for being nosied on new issues to make the dependency link back to this one for me? :) --

[issue16748] Make CPython test package discoverable

R. David Murray added the comment: I would suggest one patch and issue per test module. If multiple test modules are related enough, they could go in one patch/issue; that's a judgement call. We can make this issue dependent on those individual issues. -- ___

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: Here's version 2 of the genericpath patch. Should I try to fix everything in one patch, or one patch per test module (or group of test modules like test_(generic|mac|nt|posix)path.py)? And if separate, should each one get its own issue, or just keep them all he

[issue16748] Make CPython test package discoverable

Serhiy Storchaka added the comment: See also test_functools, test_xml_etree, test_bisect, test_bz2, test_warnings, test_decimal, test_datetime, json_tests, test_io, test_concurrent_futures, and many, many other undiscoverable tests. -- stage: committed/rejected -> patch review ___

[issue16748] Make CPython test package discoverable

Zachary Ware added the comment: Here's a patch that should clear up test_genericpath.py (and other path tests). -- resolution: fixed -> title: Ensure test discovery doesn't break for modules testing C and Python implementations -> Make CPython test package discoverable Added file: http