I don't believe that's a workable situation. At the moment, the policy is that every new intermittent failure gets a bug filed for the purpose of tracking it. There's talk (and has been for at least a year or two, now) that work will begin on OrangeFactor version 2, where intermittent failures could get tracked within OrangeFactor's own database, and we only file bugs for the failures that get too noisy, but no one has had time to work on that in the last two years, so would need prioritization from higher ups for someone(s) to dedicate some time to work on it.
KW From: e...@mozilla.com Date: Fri, 2 Sep 2016 11:35:26 -0700 Subject: Re: Reorganization of Firefox-UI tests in mozilla-central To: m...@hskupin.info CC: dev-platform@lists.mozilla.org; kwie...@gmail.com Henrik, It's not clear to me if this is just moving the location of existing tests, or if you're bringing in new tests. If it's the later, I'm concerned that we're going to get more intermittents filed in Bugzilla, which adds noise to the triage process. If you are adding more tests, then would it be possible to back off from filing intermittent bugs unless they become problematic, failing tests more than a quarter to half the time? Thanks! Emma On Thu, Sep 1, 2016 at 8:37 AM, Henrik Skupin <m...@hskupin.info> wrote: Hello, Via bug 1272145 we want to move our existing Firefox-UI tests from /testing/firefox-ui/ closer to the code which these are testing, so that the former location only contains the test harness and appropriate unit tests. Link to the current set of tests: https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests For those of you who haven't heard about these tests yet, they are basically Marionette tests with an additional layer (firefox-puppeteer) on top to ease the test creation for ui specific checks. Benefits we have are restarting and quitting the browser, and running the tests in any localized build of Firefox. As listed below you can find the current type of tests and a proposed location: * locationbar tests: /browser/base/content/test/urlbar/firefox_ui/ * private browsing tests: /browser/components/privatebrowsing/test/firefox_ui/ * safe browsing tests: /browser/components/safebrowsing/content/test/firefox_ui/ * session store tests: /browser/components/sessionstore/test/firefox_ui/ * update tests: /toolkit/mozapps/update/tests/firefox_ui/ Do those locations sound good? I have heard at least once that "firefox_ui" might not be the best choice as folder name, but that's how the harness is called, and corresponds to what we have for other harnesses too. Locations for the following tests are not clear yet: * l10n tests (shortcuts): not clear yet (maybe under /browser/base/content/test/) * security tests: not clear yet L10n related tests mostly cover shortcuts and that the correct command is invoked to find failures like bug 1173735. Nearly all of the security tests are running checks against a real server with various SSL certificates (DV, OV, EV) and protocol versions. We will have a meeting soon to determine which of those tests are needed and which ones can be removed. So we might also be able to find the correct location for the tests. For now I would like to know if the proposal is fine or if we should go a completely different route. Thanks -- Henrik _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
