On 12 Mar 2010, at 20:23, Pierre Joye wrote:
On Fri, Mar 12, 2010 at 8:10 PM, Alexey Zakhlestin
<indey...@gmail.com> wrote:
On 12.03.2010, at 22:06, Pierre Joye wrote:
Many tests fail because they are written for a given platform, or
even
worst (from a portability point of view), only for a given
configuration (library version, system version,etc.). And that's not
about windows vs other, tests can work on a Debian/ubuntu version
and
fail on another.
Well, these tests should just be removed/rewritten.
Php-tests should test php, not libraries
I would be interested to know how can we test the file API without
testing libc, for example.
But yes, that's the idea. Redesign the tests in a way that we output
something when the results of the test is not what was expected,
instead of testing random outputs from random libraries (in various
versions and flavours). No big change in run-tests (none?) but a lot
of necessary changes in existing tests.
To prefix the following comment: I've spent the previous six months
working full-time on Opera's new ECMAScript (i.e., JavaScript) engine,
and have spent a fair bit of time looking at ECMAScript test suites.
I'm somewhat doubtful about how we can ever test PHP that well: by and
large PHP-the-language is defined by PHP-the-interpreter; as long as
this is the case, how can a test have a pass/fail condition: the
interpreter is by definition correct as it is the definition of the
language you are testing?
To take an example from ECMAScript, Math.sin is defined as, "Returns
an implementation-dependent approximation to the sine of x. The
argument is expressed in radians." This is followed by a definition
which defines exact return values for the case of the argument being
±0, ±Inf, or NaN. The intention of the ECMAScript spec saying such a
thing is /exactly/ that of using system libraries: it is designed so
that you can use whatever the sin function of the maths library you
are using returns.
In the PHP manual, sin is defined as, "sin() returns the sine of the
arg parameter. The arg parameter is in radians." This is similar, but
contains one vital difference: the PHP function says it returns the
_exact_ sine (or, at the very least, does not permit any variation
from it). If the PHP interpreter is to actually follow the manual in
this case, it is pretty much impossible for it to use any system
library, as they could easily vary from the value of sine, which the
manual does not permit.
If nothing else, I think PHP testing would become a lot easier if we
had some clear specification of _what_ PHP was actually expected to
do. At the moment, all there is to go by is what is in the manual
(almost always far too vague to actually exhaustively test) and what
the implementation itself does (and it's never a good situation to be
working out what the implementation /should/ do from what it actually
does).
The first thing that needs to be done with a lot of tests that fail on
certain platforms currently within PHP is work out what result is
expected, and that is currently almost impossible for the above
reasons. I don't see how we can make any large gains in testing until
we have some clear way of knowing what behaviour is expected.
I'm certainly willing to help spec out more of PHP to the extent
necessary to be able to decently test it, if there is interest in
doing so.
--
Geoffrey Sneddon
<http://gsnedders.com/>
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
