On Wed, Jun 14, 2017 at 1:58 AM, Carsten Book <cb...@mozilla.com> wrote:
> Hi, > > we had the case that a new testsuite was enabled and one test of that > suite resulted in a new perma orange failure. Its a tier-2 testsuite but of > course also tier 2 cause additional work for sheriffs. > > The Problem on new Testsuite is that Sheriffs are in a lot of cases not > informed about that new testsuite (so we need to find out who to contact > etc) and also it affects Developers because they might be confused why > their try run is showing a orange result they don't even know of (and waste > time in finding out if that test is red/orange because of their change). > > So i propose: > > People should email with an intent to implement when adding a new > platform/suite, and that email should contain links to docs/wiki/... and > the persons (in a super optimal case more than one just for timezone > coverage if sheriffs have question) we can contact in case of questions. > > this would benefit in : > > 1) sheriffs will know who to speak to if it's failing or needs to be > hidden - so we can assign bugs or even ping people directly on irc > 2) developers know what this new test is when they break it on their try > push and so save time and resources if the failure is known and sheriffs or > others can point to a bug/newsgroup mail > 3) discussions can be had about any overlap the new tests have with other > suites or say deciding what platforms it should run on etc > > I guess this would help Sheriffs and Developers. I initially was thinking > about about a changelog on treeherder for such changes but that have the > disadvantage that this would not include all the information we need to > contact someone etc > > I hope that this proposal make sense. > There are requirements for the tiers already laid out that have not been linked in this thread: https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy I found the requirements clear but onerous. I am actively pushing a set of testsuites across the tier 2 to tier 1 boundary as we speak (for those interested in what's involved, see [1]). I was planning to get a sheriff's review of all the various pieces (documentation, failure messages, etc) by flagging an individual sheriff, but it would be better to have an intent-to-change-tier email for broader discussion. Nick [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1372075 and especially https://bugzilla.mozilla.org/show_bug.cgi?id=1370539#c3 _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
