Yesterday we were talking about enabling more extensions and adding pecl > into the mix: > http://www.mail-archive.com/internals@lists.php.net/msg54220.html > I think it would be a good idea to try to fix the currently failing tests. > http://ci.qa.php.net/job/php-src-5.3-matrix-tests/35/testReport/? > http://ci.qa.php.net/job/php-src-5.4-matrix-tests/88/testReport/? > http://ci.qa.php.net/job/php-src-trunk-matrix-tests/52/testReport/? > > There is only three failing tests left on the debian slaves: > > For 5.3 > ext/phar/tests/phar_oo_005.phpt.Phar and RecursiveDirectoryIterator 0.01 > 20 > ext/spl/tests/bug60082.phpt.Bug #60082 (100% CPU / when using references > with ArrayObject(&$ref)) > > For 5.4 and trunk > ext/standard/tests/general_functions/bug39322.phpt.Bug #39322 > (proc_terminate() loosing process resource) > > For the freebsd failures, some of them can be tackled from the test (the > locale related tests), but I see some things that need fixing in the code, > rather than in the tests: > https://bugs.php.net/bug.php?id=60186 > https://bugs.php.net/bug.php?id=60222 > > I think I also come up with a solution for the xfail issue (where do we > count those): > I would add support to run-tests to be able only run the XFAILS, or skip > them. > This way I could run the testsuite twice(either adding another axis to the > matrix config, or duplicating the php-src-$version-matrix-tests jobs in > two), and we could get a distinct report about the failing tests and the > xfails. > Having an xfail would mean a test success(in the junit.xml), having an > xfail test pass would be a fail((in the junit.xml)) but wouldn't fail the > build, only make it unstable. > This would mean that if we have failing tests ( not xfail tests, but tests > which are expected to pass failing instead) > our php-src-$version-matrix-tests build would result in a failure (red > ball). > Having only passing xfails, and no other failure would make our > php-src-$version-matrix-tests would result in an unstable build (yellow > ball). > Having no xfail tests present, and everything else passing would result > in successful and stable build (blue ball). > > -- > Ferenc Kovács > @Tyr43l - http://tyrael.hu >
Hi. A quick update: Notification mails: As you probably remember, I mentioned that currently jenkins can only send notification emails for each configuration build (debian x86, debian x86_64, freebsd x86 etc.) which isn't what we want. In the meantime I poked people on irc, and got https://issues.jenkins-ci.org/browse/JENKINS-8590 fixed. So that is a good news, it means that we can send email notifications per upstream build, not for each of the configuration builds. The bad news is that the latest email-ext plugin requires a higher jenkins version than the current LTS, to make it even worst, the soon-to-be-released next LTS will be lower than necessary. So I have to do some scripting to set up the mail notifications. I think we all agreed in the following: - we only send out mails for the aggregated results, not for each configuration build. - the commiter(s) for the build will always be notified about the result of the compilation and test build results. - if there are changes in the result of the build compared to the previous build, the list will be also notified: -- the build failed -- the build recovered -- there are "new" failing tests -- previously failing tests are passing does that sound reasonable? (some scripting also required for the test change notification) Windows: It seems that we will have some windows slaves offered by OmniTI (thanks to Elizabeth Marie Smith). FreeBSD: As I mentioned previously, the IO perf is really bad for those two vm's, so if we can't find a solution, or other place to run those, I will have to turn those off, as they really slows down the build(they are like 5-10 times slower, deleting a 100M file takes seconds, checking out the php source for a branch take literally hours). I will ran another round with the hoster, maybe we can still fix that. Other platforms: There were some suggestions that we should set up other slaves like Fedora, RHEL, etc. I think it is a good idea to support at least the major distros, but we have to still find a place to run those. Speeding up the build process: I'm also looking into other options to speed up the compilation but most importantly the testsuite builds. An obvious solution for that problem would be to not run each test in the testrun sequentially, but allow it to be executed in parallel. This isn't the first time, that this is brought up, we even have some code for it, see: https://wiki.php.net/qa/runtests http://svn.php.net/viewvc/php/phpruntests/trunk/ I plan to look into that, how far is that from the completion. I think that the current run-tests.php is too messy, so adding parallel test execution support there would require a major cleanup/refactoring, but it is possible. Improving the visualization of the test results I plan to change my current patch for run-tests.php so that the tests will be groupped into testsuites(Zend, ext/mysql, ext/session, etc.) so the All Tests list won't be that cluttered with thousands of lines as it is currently: http://ci.qa.php.net/job/php-src-5.3-matrix-tests/79/architecture=x86,os=linux-debian-6.0/testReport/? XFAIL separation: I already mentioned this, the concept of XFAILS isn't really fit into the generic test result concept, but I think that we should somehow separate those from the other tests. My idea is that the XFAILs would be logged into a separate testsuite(even separat junit.xml), so that we can handle those separately from the other test results. Support more extensions: Currently only the standard exts are compiled and tested, the only impediment extending that is currently we use the same build target for all configuration job, so if we enable a new extension, it should be available on all slave. This and the soon arriving windows slaves brings us to the next point. Different build target for each configuration job/platform. As I mentioned in the previous in the previous task, currently we use the same build target for every platform, which obviously isn't feasible on the long run. I plan to extend our ant build.xml so that it can executes a separate target based on the OS/platform, this way we can use the same job configuration in jenkins, but the platforms can have different capabilities or means to execute the build. gcov/static analysis/coverage I mentioned in the original RFC, that it would be possible to use Jenkins for static analysis also, and from the feedback it seems that many of you think that it would be a good idea. Personally I see this a little bit lower priority as the other tasks, as we have working (albeit not perfect) solutions for the static analysis stuff, but I will look into this as well, and I will poke Nuno the author/maintainer of gcov.php.net. (and static analysis tend to be pretty resource intensive so to tell the truth we don't really have the place to run those kind of jobs. yet.) adding other php projects: - PEAR: they guys at PEAR are working on fixing up the tests for PEAR, Daniel Convissor told me that I should wait that before putting that up. - Other projects: I have some ideas (Horde, ZF, ZF2, Doctrine, Propel, PHPUnit, etc.) what would make sense to add, but I would be curious what would you see to be tested against our snapshots. Thats it folks, I will update the wiki accordingly, if anybody has any question (or would like to join the efforts) feel free to shot me here or on #php.qa on efnet. ps: I was really busy recently with my day-job and other stuff, but I will try to do as much as I can, but if someone willing to help me out with some of the tasks mentioned here (or some other cool idea) please step up! :) -- Ferenc Kovács @Tyr43l - http://tyrael.hu
