On Mon, Feb 27, 2017 at 6:00 PM, Tristan Seligmann <mithra...@mithrandi.net> wrote:
> On Mon, 27 Feb 2017 at 21:54 Glyph Lefkowitz <gl...@twistedmatrix.com> > wrote: > >> That said, it has been *improving* and if it keeps improving at the rate >> it has been, I expect that we'd be able to put that coverage blocker back >> in in another 3-4 months. Perhaps something to talk about at PyCon. >> > > I think at least one problem that we're suffering from here is our fault, > rather than Codecov's: the coverage of the test suite is not stable due to > non-determinism in the test suite. That is, the lines executed during a > test run are not the same every time due to things like ordering / timing > races / etc. This means that "changes" to coverage may show up for a > particular PReven though nothing in that PR is actually responsible. > > Changes to Twisted code which are only sometimes covered by the test suite sound like they would violate a 100% coverage rule. But I guess the experience of looking at a codecov report is so bad/confusing that it's not surprising authors/reviewers might fail to see what's going on and fix the non-deterministic. Particularly for code that requires coverage measurements on multiple platforms (ie, you basically *can't* do it locally), it seems like it would be easier (though, to be clear, *bad*) to just forget about it and hope everything is covered... A tool that pointed out coverage differences between multiple runs of the same version of the code would be a useful thing to start pointing out where these flaws in the Twisted test suite lie, right? And then each area could be given deterministic test coverage instead...
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
