Today the test-verify test task will start running as a tier 2 job. Look for the "TV" symbol on treeherder, on linux-64 test platforms.
TV is intended as an "early warning system" for identifying the introduction of intermittent test failures. When a mochitest, reftest, or xpcshell test file is modified on a push, TV runs that particular test over and over until it fails (orange job, standard failure messages), or until max iterations are achieved (green job, all's well), or until TV runs out of time (green job, maybe all's well?). As a consequence, when a new test is added or a test is modified and an intermittent failure is introduced, TV will usually be the first job to fail, and it will fail on the push that modified the test, making it (usually) simple to identify where the intermittent was introduced. In future I hope to run TV on more platforms, apply it to more test suites, and refine the --verify implementation to find intermittent failures more efficiently. As a tier 2 task, TV failures will be starred but will not cause backouts. I hope to move to tier 1 once TV is proven to be effective. More info at [1]. Bug and enhancement requests welcomed: please file bugs blocking bug 1357513. [1] https://developer.mozilla.org/en-US/docs/Test_Verification _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
