Have you run 'mach bootstrap' lately?

2017-05-11 Thread Geoffrey Brown
If you set up your build environment with 'mach bootstrap' but haven't run it recently, consider taking a few minutes now to run it again. Running 'mach bootstrap' from time to time will keep your environment up to date and (more-or-less) in sync with your colleagues'. This seems to be especially

Re: Have you run 'mach bootstrap' lately?

2017-05-12 Thread Geoffrey Brown
Good idea - I filed bug 1364480. On Fri, May 12, 2017 at 8:45 AM, Sylvestre Ledru wrote: > > > Le 12/05/2017 à 05:08, Geoffrey Brown a écrit : > > If you set up your build environment with 'mach bootstrap' but haven't > run > > it recently, consider t

Re: Have you run 'mach bootstrap' lately?

2017-05-12 Thread Geoffrey Brown
I'm not sure. I always just answer the prompts and am happy with that. There is a --settings option, which sounds like it might be helpful, but I don't have any experience with that. - Geoff On Fri, May 12, 2017 at 9:00 AM, Ethan Glasser-Camp < eglasserc...@mozilla.com> wrote: > Is there a way

Re: Reminder on Try usage and infrastructure resources

2017-09-15 Thread Geoffrey Brown
Masayuki, your try push had trouble because you requested "mochitest-2" instead of "mochitest-e10s-2". Non-e10s mochitests only run on Android and Windows now. You probably wanted something like: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d68382f17d63f0674c62acc7242a9e406793895f Thi

--verify option added to mochitest, reftest, xpcshell test harnesses

2017-10-02 Thread Geoffrey Brown
The mochitest, reftest, and xpcshell test harnesses now support a --verify option. For example: mach mochitest docshell/test/test_anchor_scroll_after_document_open.html --verify In verify mode, the requested test is run multiple times, in various "modes", in hopes of quickly finding any intermi

test-verify now running as tier 2

2017-10-02 Thread Geoffrey Brown
Today the test-verify test task will start running as a tier 2 job. Look for the "TV" symbol on treeherder, on linux-64 test platforms. TV is intended as an "early warning system" for identifying the introduction of intermittent test failures. When a mochitest, reftest, or xpcshell test file is mo

Re: --verify option added to mochitest, reftest, xpcshell test harnesses

2017-10-03 Thread Geoffrey Brown
On Tue, Oct 3, 2017 at 1:05 PM, Andrew Halberstadt wrote: > This is really great Geoff! Hopefully it can cut down the number of new > intermittents we introduce to the tree. Do you know if orangefactor or > ActiveData can track the rate of new incoming intermittents? Would be neat > to see how muc

Better triage for intermittent leaks in tests?

2017-10-26 Thread Geoffrey Brown
Some of our most troublesome intermittent test failures are leak bugs ("Intermittent LeakSanitizer | leak at ..." or "Intermittent leakcheck | default process: bytes leaked ...") Even when they fail frequently, these bugs often seem to remain unresolved for many weeks. Leaks are sometimes not stro

skip-if(verify)

2018-03-06 Thread Geoffrey Brown
It is now possible to skip tests in test-verify. Simplify annotate the manifest for your test: [test] skip-if = verify or, for reftests: skip-if(verify) ... and the test-verify (TV) test task will not try to verify the annotated test. Please don't abuse this feature! Most TV failures indicate

New Android-only test suite on treeherder: geckoview-junit

2018-05-01 Thread Geoffrey Brown
With bug 1445716, there is a new Android-only, tier-1 test suite on treeherder: geckoview-junit (gv-junit). These are are on-device Android junit tests written in support of geckoview, running in our standard Android emulator environment on aws instances. You can run these tests on a local emulato

Better mach support for Firefox for Android

2015-11-30 Thread Geoffrey Brown
In recent months, many improvements have been made to mach commands to support running, testing, and debugging Firefox for Android: - More test commands for Android. These mach test commands now support Firefox for Android explicitly: mach mochitest mach robocop mach reftest mach crashte

Changes to OrangeFactor Robot comments in intermittent test failure bugs

2016-11-15 Thread Geoffrey Brown
"OrangeFactor Robot" comments in bugs for intermittent test failures now have additional information: 1. Daily and weekly comments now include the push count and failure rate. For example: 7 failures in 606 pushes (0.012 failures/push) were associated with this bug yesterday. Previously, only

Re: B2G emulator issues

2014-04-07 Thread Geoffrey Brown
On 4/7/2014 3:16 PM, Randell Jesup wrote: > The B2G emulator design is causing all sorts of problems. We just fixed That sounds very similar to some of the failures seen on the Android 2.3 emulator. Many media-related mochitests intermittently time out on the Android 2.3 emulator when run on aw

Announcing new test platform "Android 7.0 x86"

2018-11-01 Thread Geoffrey Brown
This week some familiar tier 1 test suites began running on a new test platform labelled "Android 7.0 x86" on treeherder. Only a few test suites are running so far; more are planned. Like the existing "Android 4.2" and "Android 4.3" test platforms, these tests run in an Android emulator running in

PSA: Android test environments

2019-10-18 Thread Geoffrey Brown
The Android test environments used for continuous integration have been through many changes over the last year or two; here's a review of what we have today. [1] Most of our Android tests run on emulators. Some run on hardware: real phones. Our Android hardware tests run on physical devices -- M

Re: Visibility of disabled tests

2020-01-11 Thread Geoffrey Brown
Thanks Johann. I agree it is important that we try to fix tests that have been disabled. I think the sheriffs usually needinfo the triage owner before/when disabling a test; I'm disappointed to hear that isn't happening consistently. However, I'd prefer not to change the review process for the dis

Android tests - do not forget logcat!

2020-02-14 Thread Geoffrey Brown via dev-platform
Lately it feels like there is more activity around running, investigating, and fixing Android automated tests -- that's great to see! When looking at logs from automated test logs, be aware that there are differences between desktop and Android. Some messages dumped to standard output will appear