New submission from Paul Ganssle <p.gans...@gmail.com>:
Following up on this year's language summit, I would like to propose that we start integrating property tests into the standard library. Zac Hatfield-Dodds (along with myself and Carl Friedrich Bolz-Tereick) have put together this repository of tests that we run against the standard library as is: https://github.com/Zac-HD/stdlib-property-tests Here is the blog post covering the proposal from the language summit: https://pyfound.blogspot.com/2020/05/property-based-testing-for-python.html The biggest challenges here are logistical: 1. Pulling in third party dependencies is difficult to do reliably on CI, but checking in hypothesis's entire dependency tree is probably not feasible. 2. We don't necessarily want to require developers running their tests locally to have to set up a hypothesis environment just to run the tests. 3. Hypothesis tests are not (by default) deterministic, which some are concerned may lead to flakiness by themselves. To allay these concerns, I propose that we implement a compatibility interface for hypothesis that uses the third party module when it's installed, but otherwise falls back to stubs. The way I see the stubs working is that `@given` *without* `@example`s would simply skip the test. If you specify `@given` and one or more `@example`s, the test falls back to a simple parameterized test when hypothesis is not available. At least at first, we won't attempt to add a mandatory PR job that runs with hypothesis installed. Instead, I'd like to run either an optional job on PR or have one or more buildbots that runs the hypothesis tests. I would also like to suggest a policy of including at least one example in each property test, so that on PR at least some of the inputs are tested. ---------- assignee: p-ganssle components: Tests messages: 379226 nosy: Zac Hatfield-Dodds, p-ganssle, terry.reedy priority: normal severity: normal stage: patch review status: open title: Use hypothesis for testing the standard library, falling back to stubs type: enhancement versions: Python 3.10 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue42109> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
