On 3/29/12 3:00 PM, Alexey Shein wrote:
Hi, internals! I've got a suggestion about refactoring our tests suite. I'd like to remove XFAIL institution and mark all failing tests just as FAIL. XFAIL has a problem that it hides attention from failing tests depending on not yet fixed bugs (most important), not yet implemented features (less important). Failed tests should make pain. They should bug you every day until you go and fix them. XFAILs serve now as a pain-killers, we've got about 50 of them in the repo, so devs (I assume) think this way: "It's failing, but it's EXPECTED to fail, so let's leave it as is". That's wrong thinking. Either tests are correct and if they fail you should fix the code and leave them failed until the code is fixed, or, if the tests are incorrect - fix the tests or remove them completely.
The XFAIL mechanism reflects the reality of open source that not all bugs are fixed. We need a simple, low maintenance way to have a 'clean' testsuite shipped which exhibits minimal noise so that users don't waste time investigating known failures. XFAIL also allows end users to see when something has broken that used to work. If the system is being overused, feel free to call people out on it. I don't think it should be used for unimplemented features long term. XFAIL is a simple mechanism. Anything different like moving tests to a special 'failed' directory adds burden. I don't belive we have extra cycles for this, but would be happy to be proved wrong. Chris -- Email: christopher.jo...@oracle.com Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
