With the advent of gabbi tests in both ceilometer and gnocchi, we've started using xfail (expected failure) as a way of highlighting HTTP behavior that is wrong or poor[1] and linking to bugs on launchpad in the description of the tests.
This means that we need to start monitoring local test runs for unexpected success to see which of these have been fixed, and update the tests accordingly. testr itself will not alert you. Gnocchi has already started using pretty_tox.sh, so as long as there is a recent version of subunit-trace (from tempest-lib) installed, gnocchi test runs will tell when there has been an unexpected success. Ceilometer hasn't made that change, but it is possible to do it by hand. After any test run: testr last --subunit |subunit-trace will report on the most recent test run and give a summary. Again, this assume a recent tempest-lib has been installed (from PyPI). [1] Often but not always related to the framework being used. Usually, but not always, WSME failing to trap exceptions in the face of unexpected data. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
