Hi, we had the case that a new testsuite was enabled and one test of that suite resulted in a new perma orange failure. Its a tier-2 testsuite but of course also tier 2 cause additional work for sheriffs.
The Problem on new Testsuite is that Sheriffs are in a lot of cases not informed about that new testsuite (so we need to find out who to contact etc) and also it affects Developers because they might be confused why their try run is showing a orange result they don't even know of (and waste time in finding out if that test is red/orange because of their change). So i propose: People should email with an intent to implement when adding a new platform/suite, and that email should contain links to docs/wiki/... and the persons (in a super optimal case more than one just for timezone coverage if sheriffs have question) we can contact in case of questions. this would benefit in : 1) sheriffs will know who to speak to if it's failing or needs to be hidden - so we can assign bugs or even ping people directly on irc 2) developers know what this new test is when they break it on their try push and so save time and resources if the failure is known and sheriffs or others can point to a bug/newsgroup mail 3) discussions can be had about any overlap the new tests have with other suites or say deciding what platforms it should run on etc I guess this would help Sheriffs and Developers. I initially was thinking about about a changelog on treeherder for such changes but that have the disadvantage that this would not include all the information we need to contact someone etc I hope that this proposal make sense. Cheers, - Tomcat _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
