Hi!
- According to this website there are still 94 test failures in 5.4 . Can you confirm all of them are minor problems? http://gcov.php.net/viewer.php?version=PHP_5_4
Most of them appear so, I'll go through them again to be sure and encourage others to do so too and raise red flags if somebody sees something bad there. Unfortunately, some tests are environment-dependent or otherwise have subtle dependencies or structure that make them work on one system and fail on another not because of the bug in PHP but because of the test itself. So, I have 0 fails on my Linux build but 6 fails on my Mac build. Other times some systems may not support some capability, use old version of the library, etc. and the test may not account for that.
I do not think it is practical to postpone release until we solve all of such problems, since this being volunteer-driven open-source project this means not having any release schedule at all. I prefer having the schedule even if that means we'd have to release with some known deficiencies.
- There was this problem with 5.3.7 and the crypt() bug. Has there been some improvement in the process of handling test failures? For example mark expected failures as expected failures, and fix the tests or the code? Or are the failing tests "stable" since month and all of them are minor problems?
Yes, there was work done on these. Most of those were fixed, but few still remain, especially across various environments (i.e. test may be fine on some but not others). I of course am all for fixing that further and welcome any help on that.
- There have been 319 unique failed tests in RC5, reported by user tests. Is someone looking into them and trying to classify and/or fix them? http://qa.php.net/reports/
Non-reproducible failures usually mean the problem is with the test itself, or with the difference of expectations in the test and local environment, not with PHP. It may still be PHP problem, of course, so the person running the test should check it out and submit a bug if appropriate and if it's bad enough, also send a message to this list.
All in all the number of test failures still feels very high, I would be interested in your opinion. Is this "normal" for big projects like this?
I do think it should be reduced, however if the choice is between waiting forever and have release with some bugs, I think it is practical to choose the latter. Of course, if we discover a major problem that makes PHP unusable or seriously impacts many PHP users, it will be dealt with immediately, and had been so in the past, but otherwise we have to work within the realities of a big project with limited resources and realize that while we strive for 0 bugs in every release, it may never be possible.
-- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
