Something we've been missing for a long time is high level visibility
into the state of our tests. How many tests from suite X are we skipping
on platform Y?
A new project called Test Informant [1] is keeping track of this data
and allows us to generate reports with it. Here is a report generated
from yesterday's builds:
http://people.mozilla.org/~ahalberstadt/informant-reports/daily/2014-09-29.informant-report.html
For more details including known limitations and future work, see my
blog post on the topic:
http://ahal.ca/blog/2014/how-many-tests-are-disabled/
I intend to automate the process of generating both daily and weekly
reports (daily reports will show how many tests were enabled/disabled
from one day to the next, weekly reports from one week to the next). I'd
like to post an automated weekly report to dev.platform starting next week.
Please let me know if this sounds reasonable, if you have
feedback/questions or if you notice any inconsistencies with the report
and the real world state of manifests.
Cheers,
Andrew
[1] https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
