Re: Project Stockwell - February 2017 update

Just noticed no one looped back here. Joel filed bug 1337844 and bug 1337839 . There has been some discussion there. To summarize, running tests locally is currently optimized towards "Run a

Re: Sheriff Highlights and Summary in February 2017

I don't have any data to back this up, but my suspicion is that a large percentage of backouts had try runs, but said try runs didn't run the jobs that failed and caused the backout. Imo, these kinds of backouts are (more) acceptable because it means the developer was trying to avoid doing a full t

Errors found by cppcheck

I was doing a bit of research into cppcheck [1] to see if it might be worth running as a linter (alongside eslint, flake8 etc). More discussion in bug 1370292 [2]. I ran it against the entire tree [3] and got these results: https://bug1370292.bmoattachments.org/attachment.cgi?id=8874498 So far it

Re: Errors found by cppcheck

> checks that rare valuable ones to have turned on by default, and leave the > rest off by default. (I'm hoping the effectiveness of its check isn't > context sensitive...) > > > > On 06/06/2017 09:06 AM, Andrew Halberstadt wrote: > >> I was doing a bit o

Linting for common causes of oranges in mochitests (need ideas)

I'm looking into creating custom eslint rules that aim to avoid common causes of oranges in tests. We have an MDN page containing some of these already. Some of those patterns might be pretty hard to catch with a li

Re: Linting for common causes of oranges in mochitests (need ideas)

> > > On 7/6/17 11:47 AM, Andrew Halberstadt wrote: > > - Are there additional things not listed on that page that we could lint > for? > > > Do we want to discourage tests from using Date (`new Date` or > `Date.now()`) for measuring time? Dates are affected by time zon

Re: Linting for common causes of oranges in mochitests (need ideas)

On Fri, Jul 7, 2017 at 12:15 PM, Ehsan Akhgari wrote: > FWIW years ago I decided to act on trying to detect some of the patterns > indicated on that page automatically, see https://bugzilla.mozilla.org/s > how_bug.cgi?id=649012. It didn't use linting (because we didn't really > have any linting

ESlint rule 'no-arbitrary-setTimeout' enabled on xpcshell tests

As part of a larger effort to reduce oranges, we are starting to lint our tests for common causes of intermittent failures. One low-hanging fruit is preventing setTimeout with an arbitrary value (aka non-zero) as opposed to waiting for an appropriate event. The mochitest harness already prevents th

Re: ESlint rule 'no-arbitrary-setTimeout' enabled on xpcshell tests

ble to use non-0 > timeouts. > > Cheers, > Felipe > > On Fri, Jul 28, 2017 at 12:48 PM, Andrew Halberstadt < > ahalberst...@mozilla.com> wrote: > >> As part of a larger effort to reduce oranges, we are starting to lint our >> tests for common causes of intermit

./mach try fuzzy: A Try Syntax Alternative

I'm pleased to announce an alternative method for scheduling tasks on try is now landed on mozilla-central. It makes use of the awesome fzf [1] project to filter down the list of all task labels with a fuzzy matching algorithm. It works both with mercurial or git. If using mercurial, you'll need t

Re: reducing high try macosx pending counts

Using the new |mach try fuzzy| selector will also make it a lot easier to only schedule exactly what you need. To run what the above try syntax uses, do: $ ./mach try fuzzy !osx 'web-platform That will run every task that doesn't contain the string 'osx', and does contain the string 'web-platfor

Re: ESlint rule 'no-arbitrary-setTimeout' enabled on xpcshell tests

This is now also enabled on browser-chrome tests. Bug 1389234 has been filed to track deprecating SimpleTest.requestFlakyTimeout on mochitest-plain and chrome in favour of this new rule. -Andrew On Fri, Jul 28, 2017 at 12:02 PM Andrew Halberstadt < ahalberst...@mozilla.com> wrote: > Ah

PSA: Set prefs in mochitest manifests

Bug 1328830 recently landed and has added the ability to set prefs directly in a mochitest manifest. Prefs can be set like this: [DEFAULT] prefs = browser.newtabpage.introShown=true layout.css.servo.enabled=true [browser_foo.js] [brow

Re: disabled non-e10s tests on trunk

We now have the ability to set prefs from a mochitest manifest (see bug 1328830 and my recent newsgroup post). We could refactor these tests into a special non-e10s manifest that sets browser.tabs.remote.autostart=false and keep running them as

Re: ./mach try fuzzy: A Try Syntax Alternative

h -q so it's in your shell history) -Andrew On Wed, Aug 2, 2017 at 10:30 AM Andrew Halberstadt wrote: > I'm pleased to announce an alternative method for scheduling tasks on try > is now landed on mozilla-central. It makes use of the awesome fzf [1] > project to filter down t

Re: ./mach try fuzzy: A Try Syntax Alternative

artifact builds. There are bugs on file to reach parity here. On Wed, Aug 2, 2017 at 10:30 AM Andrew Halberstadt wrote: > I'm pleased to announce an alternative method for scheduling tasks on try > is now landed on mozilla-central. It makes use of the awesome fzf [1] > project t

Re: ./mach try fuzzy: A Try Syntax Alternative

On Sat, Sep 2, 2017 at 1:00 PM Randell Jesup wrote: > >+to...@lists.mozilla.org > > > >There have been a bunch of new features added here I'd like to highlight: > > > - --env: You can now set environment variables in your tasks directly > > from the command line, e.g: > > - ./mach try fu

Re: Intermittent oranges and when to disable the related test case - a simplified policy

On Fri, Sep 8, 2017 at 7:10 PM Gregory Szorc wrote: > I know we've topic in this topic in the past but I can't recall outcomes. > Is it worthwhile to define and use a richer test manifest "schema" that > will facilitate querying and building dashboards so we have better > visibility into disabled

Re: Intermittent oranges and when to disable the related test case - a simplified policy

On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan wrote: > On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt < > ahalberst...@mozilla.com> wrote: > >> I don't think so, that data already exists and is query-able from >> ActiveData: >> https://activ

Re: Reminder on Try usage and infrastructure resources

There's sort of a way to do this with try syntax. I say sort of because it doesn't support all suites and there seems to be a few bugs with it. But you can try: ./mach try -b o -p linux64 -u none path/to/dir/or/test This should only run the directory or test you specified (it'll always show up as

Re: Reminder on Try usage and infrastructure resources

Yes, all mochitests except Android restart between manifests (which is usually the same as folders). On Thu, Sep 14, 2017 at 12:03 PM Marco Bonardo wrote: > On Thu, Sep 14, 2017 at 5:56 PM, James Graham > wrote: > > On 14/09/17 16:48, Marco Bonardo wrote: > > mach try -p linux64 > > Afaict, th

Re: Intent to require `mach try` for submitting to Try

Yes, we'd like to make it the default eventually. I've been holding off for now, but expect it will be switched to the default at some point in Q4 or Q1. If you don't want to wait and are tired of typing 'fuzzy' each time, you can create a ~/.mozbuild/machrc file and add: [try] default = fuzzy On

Re: Intent to require `mach try` for submitting to Try

Interesting, please file a bug either way and CC me. It worked for me back when it first landed, though I haven't tried it since. I'll see if I can reproduce sometime this week. On Sat, Sep 16, 2017 at 4:31 AM Marco Bonardo wrote: > I have a problem with try fuzzy, not sure if it's just my syste

Re: --verify option added to mochitest, reftest, xpcshell test harnesses

This is really great Geoff! Hopefully it can cut down the number of new intermittents we introduce to the tree. Do you know if orangefactor or ActiveData can track the rate of new incoming intermittents? Would be neat to see how much of an impact this tool has on that. On Mon, Oct 2, 2017 at 1:08

Re: Intent to Enable: Automated Static Analysis feedback in MozReview

On Tue, Oct 17, 2017 at 4:00 PM wrote: > Those things can be encoded as regexps, but they span across programming > languages (XUL, XHTML, HTML, XBL, DTD, JS, C++). > To create a new regex-based linter, simply add a file like this: https://dxr.mozilla.org/mozilla-central/source/tools/lint/test-d

Re: Intent to require Python 3 to build Firefox 59 and later

On Fri, Nov 10, 2017 at 9:44 PM David Burns wrote: > My only concern about this is how local developer environments are going > to be when it comes to testing. While I am sympathetic to moving to python > 3 we need to make sure that all the test harnesses have been moved over and > this is someth

Re: overly strict eslint rules

Not replying to anyone specifically, but Sylvestre's team is working on getting our linters hooked up to mozreview/phabricator as well. I think this paired with bootstrapping the hooks will smooth out the lint fixing workflow considerably. Andrew On Fri, Jan 5, 2018 at 5:58 AM Mark Banner wrote

PSA: Default log format changed for testing commands

You should notice a new colourized log format when running tests with |mach test| or |mach mochitest|. If you want to revert to the old log format you can add: [test] format=tbpl to your ~/.mozbuild/machrc. If you have any feedback or issues with the new format, please chime in to bug 1445624

Re: Removing tinderbox-builds from archive.mozilla.org

Going back to the original question, it looks like mozregression doesn't use the builds that Nick wants to remove anyway. So regardless of our retention policies, it looks like removing these builds would have no impact on mozregression's effectiveness. Is that an accurate statement? -Andrew On W

PSA: Setting preferences and extensions in test harnesses

tl;dr - You can now set prefs and install extensions across multiple harnesses by modifying the relevant profile under testing/profiles. If you previously would have set a pref in: testing/profiles/prefs_general.js, Use this instead: testing/profiles/unittest/user.js # Overview I'm currently

Re: PSA: Setting preferences and extensions in test harnesses

On Wed, May 16, 2018 at 11:07 AM Andreas Tolfsen wrote: > Any plans to consolidate the Mn preferences, currently stored in > geckoinstance.py? > Yes, but I don't have a timeline. I want to at least finish up reftest and xpcshell first. Then time permitting, I'd like to finish up marionette, cppu

Re: OrangeFactor/Intermittent Failures View Update

ew On Wed, Jun 6, 2018 at 8:34 AM Andrew Halberstadt wrote: > Thanks for all your work here, this looks much cleaner and easier to > interpret. Looking forward to the future improvements you mentioned! > > For anyone bad at remembering URLs, you can get to this view by clicking > the

Re: PSA: pay attention when setting multiple reviewers in Phabricator

It might be worth investigating whether we can switch Phabricator's default (so that multiple reviews are all blocking, and to make them non-blocking would require the extra step). Personally I think setting multiple reviewers up on a first come first serve is disrespectful to those reviewers' time

Re: PSA: Automated code analysis now also in Phabricator

Congrats, thanks to everyone involved for getting this working! I really like the idea of comparing errors with and without the patch, this would let us lint code where linting isn't explicitly enabled in mach lint/CI. One caveat to doing is that the review bot would need to make it very clear whi

PSA: Re-run old (non-syntax) try pushes with |mach try again|

While |mach try fuzzy| is generally a better experience than try syntax, there are a few cases where it can be annoying. One common case was when you selected a bunch of tasks in the interface and pushed. Then at a later date you wanted to push the exact same set of tasks again. This used to be a r

Re: ./mach try fuzzy: A Try Syntax Alternative

I recently added the ability to specify --query multiple times (where the set of tasks is the union of each individual query). So something like: ./mach try fuzzy -q "'android !pgo !cov" -q "'build !pgo !cov" Should also accomplish what you want. It's still a bit clunky as multiple queries don't

Re: PSA: Local static analysis builds on Linux and Mac OS X

On 15/10/15 01:59 PM, Ehsan Akhgari wrote: On 2015-10-14 9:18 PM, Mike Hommey wrote: BTW, since we're growing the set of checks the plugin does, what do we do to ensure that it doesn't add too much overhead to the compilation time? Currently nothing beyond taking care to not add slow checks.

Re: Merging comm-central into mozilla-central

On 23/10/15 04:43 AM, Gregory Szorc wrote: - It adds burden to developers, needing to support those projects merged from comm-central. Just look around in mozilla-central at all the optional things that are not visible on treeherder and break regularly. The situation wouldn't b

Re: Now measuring Firefox size per-commit. What else should we be tracking?

ActiveData is just a very large data base. An automated-client would be something that periodically runs a query, formats the data and plugs it into a graph. Here's an example of a client side JS tool that runs a query to determine which tests are enabled or skipped: https://github.com/mozilla/tes

Re: mozregression 2.0.0

Just want to point out Julien also added a mach command wrapper for mozregression that installs it and everything. Just add 'mach' in front of his examples to try it out. Thanks for all the work here Julien! On 02/12/15 04:42 PM, Julien Pagès wrote: Hello, I'm happy to announce the release 2.0

Re: Too many oranges!

On 22/12/15 11:38 AM, Ben Kelly wrote: On Tue, Dec 22, 2015 at 11:16 AM, Douglas Turner wrote: Mike -- totally supportive of this. I would *love* to see a release cycle completely dedicated to quality. We branch again on January 26. We could use that cycle to focus on nothing but quality (fi

Re: Too many oranges!

+1 I'd prefer to see quality be a perpetually ongoing effort over a periodic burst. I'd like to see management rotate people into "quality mode" by explicitly setting goals and deliverables around addressing it. The problem in the past has been this notion of "greatest impact". Things like refac

Re: e10s tests

I've been a little hesitant to plug this as it's a prototype and the UX is pretty terrible. Also the underlying database it's using is not super resilient. But people might find it useful for this effort and I don't have time to improve it much more anyway, so what the heck. This is basically wha

Re: Changes to the firefoxtree hg extension

Thanks for sharing this, I've seen a few people asking about it. Note, as of bug 1234927 the `wip` alias that |mach mercurial-setup| provides you has been fixed. But if you already had a pre-existing `wip` alias, you'll have to update it manually like Patrick mentions. -Andrew On 11/01/16 06:00

Re: ESLint checks are now running on the nightly trees on checkin

On 14/01/16 12:24 PM, Dave Townsend wrote: All hail the mighty releng folks who have made it so ridiculously easy to prototype and deploy new taskcluster tests! I just want to emphasize the significance here. A developer unaffiliated with either releng or the ateam and with only minimal support

Re: Just Autoland It

On 22/01/16 12:20 AM, Nicholas Nethercote wrote: On Thu, Jan 21, 2016 at 6:35 PM, Gregory Szorc wrote: I've gotten into the habit of just landing things if I r+ them and I think they are ready to land. This has startled a few people because it is a major role reversal of how we've done things

Re: Just Autoland It

On 22/01/16 02:12 AM, Gregory Szorc wrote: On Thu, Jan 21, 2016 at 10:37 PM, Mike Connor wrote: Like Greg, I'm a big fan of reviewer-lands-if-ready. It's a huge simplification of workflow, saves developers time, and lets machines do work instead of humans. That said, I don't think we should be

Re: Just Autoland It

On 22/01/16 10:30 AM, Boris Zbarsky wrote: On 1/22/16 10:08 AM, Andrew Halberstadt wrote: In the end, it's on the reviewer. If the patch is complicated, has open dependencies or looks like it might cause problems, as a reviewer I'm not going to land it no matter what. If it's

Re: Moving FirefoxOS into Tier 3 support

On 25/01/16 11:00 AM, Fabrice Desré wrote: We're also working on a solution to move the b2g builds & tests to their own infrastructure from which we'll ship our builds & updates. What specifically does this mean? Do you mean infrastructure at the IT level? Or at the continuous integration le

Re: Just Autoland It

On 25/01/16 05:44 PM, Eric Rescorla wrote: On Mon, Jan 25, 2016 at 1:58 PM, Mike Hommey wrote: On Mon, Jan 25, 2016 at 11:23:59AM -0800, Steve Fink wrote: Heh. Your list of UI complaints is very similar to mine. Some comments: On 01/25/2016 04:26 AM, Honza Bambas wrote: Writing both as a p

Re: Moving FirefoxOS into Tier 3 support

Hi, I'm on the engineering productivity team, and work a lot on continuous integration and test harnesses. I stood up initial Gecko unittests on B2G emulator, worked on B2G automation for several years up until the divide, and still help nominally maintain the emulator and mulet unittests. So that

Re: Just Autoland It

On 28/01/16 06:31 PM, Eric Rescorla wrote: On Thu, Jan 28, 2016 at 10:58 AM, Gregory Szorc wrote: I'd like to thank everyone for the feedback in this thread. However, the thread has grown quite long and has detoured from its original subject. Speaking on behalf of everyone who works on MozRev

Reftests moving to structured logging

Reftest is the last major test harness still not using structured logs, but that should change by the end of the week. See bug 1034290 [1] for more details. I've tried my best to make sure things like reftest-analyzer, leak/assertion checks, crash detection, etc. all continue to work. But due to

Re: Reftests moving to structured logging

This is now live on central. On 04/02/16 01:28 PM, Andrew Halberstadt wrote: Reftest is the last major test harness still not using structured logs, but that should change by the end of the week. See bug 1034290 [1] for more details. I've tried my best to make sure things like reftest-ana

Re: Generic Task Cluster Tasks / File Based Task Scheduling

On 17/02/16 01:59 PM, Gregory Szorc wrote: * You can now only run tasks if certain files changed. This opens the door for drastic reduction in automation load via more intelligent scheduling of tasks via in-tree configurations. (For the curious, relevant commits/files are determined from a combin

Test automation addons and signing

To date, our continuous integration has been setting 'xpinstall.signatures.required=false' to bypass addon signing. But soon, this pref will become obsolete and Firefox will enforce signing no matter what. In preparation, we will begin signing extensions that are used in our test automation. If y

Re: Exit code -11 must die

On 29/02/16 12:03 PM, Benjamin Smedberg wrote: On 2/27/2016 9:06 PM, Randell Jesup wrote: months until recently it popped up a bit). Note that this failure *never* results in a crashdump, and I've never seen it locally, just in Automation. What we do know: * Exit code -11 is evidence a SIG

Re: firefox-ui-tests are now in mozilla-central with TaskCluster support

They're more like end-to-end tests than browser-chrome. E.g instead of calling Gecko APIs, they'll send a mouse event to a button, send key events to a textbox, etc. So they'll theoretically perform actions closer to what a user might do. Since they're written in python, they can also do things l

Try syntax is no longer required when pushing to try

With treeherder's "Add New Jobs" [1] UI, using try syntax is no longer the only way to schedule stuff on try. As of now, it's possible to push to try without any try syntax. If you do this, no jobs will be scheduled on your push and it will be up to you to add whichever jobs you want via "Add New

Re: Try syntax is no longer required when pushing to try

On 03/03/16 12:37 PM, Andreas Tolfsen wrote: On 3 March 2016 at 17:25, Andrew Halberstadt wrote: With treeherder's "Add New Jobs" [1] UI, using try syntax is no longer the only way to schedule stuff on try. As of now, it's possible to push to try without any try syntax. If

Intent to enable e10s by default when running tests locally

Bug 1243083 tracks enabling e10s by default when running tests locally. I intend to do this for as many harnesses as possible unless someone says otherwise. The implication is that the --e10s flag will be renamed to --disable-e10s. So you'll need to pass in --disable-e10s if you wish to run witho

Re: Intent to enable e10s by default when running tests locally

ll be able to make a .machrc with: [defaults] mochitest = --disable-e10s On 24/03/16 12:30 PM, Boris Zbarsky wrote: On 3/24/16 12:15 PM, Andrew Halberstadt wrote: If you have any concerns or know of other suites that should explicitly *not* be run with e10s enabled by default, please let me kno

Re: Intent to enable e10s by default when running tests locally

sn't a big deal. On 24/03/16 01:25 PM, Andrew McCreight wrote: On Thu, Mar 24, 2016 at 9:15 AM, Andrew Halberstadt < ahalberst...@mozilla.com> wrote: Bug 1243083 tracks enabling e10s by default when running tests locally. I intend to do this for as many harnesses as possible unless s

Re: Intent to enable e10s by default when running tests locally

switch to enable running under both modes (or even better, defaulting to both modes). - Blair Andrew Halberstadt wrote: Bug 1243083 tracks enabling e10s by default when running tests locally. I intend to do this for as many harnesses as possible unless someone says otherwise. The implicatio

Re: Intent to enable e10s by default when running tests locally

he ergonomics around testing both to be quite a footgun - it's a manual process to run tests under both modes, and it's too easy to forget to run tests under the other mode. Would be handy to have a switch to enable running under both modes (or even better, defaulting to both modes). - Blai

Re: Intent to enable e10s by default when running tests locally

. There is no eta at this time. -Andrew On 24/03/16 12:15 PM, Andrew Halberstadt wrote: Bug 1243083 tracks enabling e10s by default when running tests locally. I intend to do this for as many harnesses as possible unless someone says otherwise. The implication is that the --e10s flag will be

Mach command aliases and runtime configuration

Hey all, bug 1255450 has landed which means.. For mach users: You can now create a machrc (or .machrc) file in the following locations: * $MACHRC * ~/.mozbuild * topsrcdir In the future, individual commands may implement their own settings, but for now a single section called 'alias' is implemen

Re: One Firefox repository to rule them all

This is really cool! Though I much prefer firefoxtree's namespace updating to keep track of remote heads over using bookmarks. I want a label that will always point to the last known head on the server, so e.g `hg update central && hg commit -m "Foo"` should not move 'central'. Using bookmarks to

Re: Searchfox (new code search tool)

On 07/06/16 01:18 AM, Mike Hommey wrote: I'm going to sound negative, but why? Or more precisely, why not contribute to DXR to add those features that you implemented in searchfox that DXR doesn't have? MXR is already taking too long to fade out of existence, do we really want yet another differ

ESlint infrastructure moved to 'tools/lint'

If you had any absolute paths in editor configuration, you'll need to update them to point to 'tools/lint/eslint'. This change was in preparation for getting |mach eslint| integrated with the mozlint library, see bug 1258341 for details. My aim will be to maintain backwards compatibility both in

Re: Rust required to build Gecko

When first installing rust with ./mach bootstrap the install is successful, but there is a message about not being able to find the compiler immediately afterwards. For anyone confused by this, the binaries are downloaded to ~/.cargo/bin and adding this directory to your $PATH should fix the issu

Several leak failures have slipped passed continuous integration

Over the holidays, we noticed that leaks in mochitest and reftest were not turning jobs orange, and that the test harnesses had been running in that state for quite some time. During this time several leak related test failures have landed, which can be tracked with this dependency tree: https:

Re: Several leak failures have slipped passed continuous integration

Ah, I misunderstood. If the log line is printed or dumped directly to stdout, then the string TEST-UNEXPECTED-FAIL actually will fail the job. It's only if we do log.info('TEST-UNEXPECTED-FAIL ...') that we run into problems. So if you see the string 'TEST-UNEXPECTED-FAIL' somewhere, it isn't nec

Test harness preferences moving out of automation.py

tl;dr - If you update a test harness pref in build/automation.py.in, please also update it in testing/profiles/prefs_general.js. This is temporary and I am working to remove the automation.py.in location in bug 830430[1] --- Currently prefs that are common to our test harnesses are hardcoded

Test harness preferences moved out of build/automation.py.in to testing/profiles/prefs_general.js

As part of a broader automation refactor, we are consolidating all of the test harness preferences in one place. Storing them as a string in a python file is less than ideal, so as a first step the preferences in automation.py.in have been moved to testing/profiles/prefs_general.js [1]. Note t

Re: Embracing git usage for Firefox/Gecko development?

I used to also really dislike mercurial, but the more I used it the more I started to realize its power. Gps summarized it much better than I ever could have in a blog post last month: http://gregoryszorc.com/blog/2013/05/12/thoughts-on-mercurial-%28and-git%29/ I think mercurial and git are bo

Mach for b2g emulator unittests

Running mochitests/reftests/xpcshell on a B2G emulator has been a bit of a struggle in the past. As of now, if you have an emulator configured and built, you can simply run './mach mochitest-remote' (or ./mach help mochitest-remote for additional arguments). Substitute 'mochitest' with 'reftest

Re: Poll: What do you need in MXR/DXR?

Better python support. For example, the function name parameter doesn't work with ext: .py http://dxr.mozilla.org/mozilla-central/search?q=function%3Astart%20ext%3A.py <-- no results http://dxr.mozilla.org/mozilla-central/search?q=%22def%20start%22%20ext%3A.py <-- results On 10/02/2013 03:

Defining a standard structured logging format for test harnesses

tl;dr if you have opinions on a standardized structured logging format for test harnesses, please comment on https://bugzilla.mozilla.org/show_bug.cgi?id=916260 Hi all, our past intern Chris Manchester worked on implementing structured logging for our test harnesses this summer [1]. He wrote

Re: Pushes to Backouts on Mozilla Inbound

On 11/05/2013 01:49 PM, Chris Peterson wrote: On 11/5/13, 7:10 AM, James Graham wrote: Wht data do we currently have about why the wait time is so long? If this data doesn't exist, can we start to collect it? Are there easy wins to be had, or do we need to think about restructuring the way that

Re: Reftests execute differently on Android or b2g?

On 01/07/2014 07:23 PM, Neil wrote: I tried to check in a reftest today. Apparently it fails on Android and b2g. The failure mode appears to be that the reftest takes a screenshot before the test has loaded (the page is still blank, whereas it should have a red square for failure or a green squar

Re: Splitting of tests in the mochitest suite

A downside to chunking such that skipped tests are taken into account is that it will make some chunks take longer to run than others. This negatively impacts the scheduling algorithms and ultimately increases infrastructure load. The effects are especially noticeable on new platforms/applicat

Re: Splitting of tests in the mochitest suite

On 01/27/2014 03:36 PM, Ehsan Akhgari wrote: On 1/27/2014, 7:44 AM, Andrew Halberstadt wrote: A downside to chunking such that skipped tests are taken into account is that it will make some chunks take longer to run than others. This negatively impacts the scheduling algorithms and ultimately

Re: Tech reviewer needed for FxOS reftests article

Hey Chris, I can do a review. Though fair warning, I wrote most of it :). It looks like it's still accurate, though I would probably delete everything other than how to run them using mach. I don't think there is any reason to support the old manual way of doing it anymore. Let me know what yo

Test harness options now defined in-tree -- please pull m-c before pushing to try!

With the completion of bug 981030 test harness options are now defined in-tree. This means any push without this patch [1] will fail. I already went ahead and landed it on all release branches, but if you have an integration or project branch, you may need to merge m-c in (or cherry pick that

Re: Test harness options now defined in-tree -- please pull m-c before pushing to try!

On 26/03/14 12:19 AM, Gregory Szorc wrote: Andrew: I'm curious if you or anyone else has given any consideration into making it dead simple to change the config so only a subset of tests will be active. e.g. I'd like to save automation resources and filter my browser chrome jobs to tests under to

Re: Test harness options now defined in-tree -- please pull m-c before pushing to try!

On 26/03/14 09:06 AM, Andrew Halberstadt wrote: On 26/03/14 12:19 AM, Gregory Szorc wrote: Andrew: I'm curious if you or anyone else has given any consideration into making it dead simple to change the config so only a subset of tests will be active. e.g. I'd like to save automation

Re: Test harness options now defined in-tree -- please pull m-c before pushing to try!

On 26/03/14 01:45 PM, Nathan Froyd wrote: - Original Message - Releng (and I tend to agree with them) is very opposed to having one config affect multiple platforms (or at least multiple buildapps). Over time it tends to lead to messy and hard to follow configs. It also makes it harder i

Re: Policy for disabling tests which run on TBPL

On 06/04/14 08:59 AM, Aryeh Gregor wrote: On Sat, Apr 5, 2014 at 12:00 AM, Ehsan Akhgari wrote: Note that is only accurate to a certain point. There are other things which we can do to guesswork our way out of the situation for Autoland, but of course they're resource/time intensive (basically

Re: Policy for disabling tests which run on TBPL

On 04/04/14 03:44 PM, Ehsan Akhgari wrote: On 2014-04-04, 3:12 PM, L. David Baron wrote: Are you talking about newly-added tests, or tests that have been passing for a long time and recently started failing? In the latter case, the burden should fall on the regressing patch, and the regressing

Re: Policy for disabling tests which run on TBPL

On 07/04/14 05:10 AM, James Graham wrote: On 07/04/14 04:33, Andrew Halberstadt wrote: On 06/04/14 08:59 AM, Aryeh Gregor wrote: Is there any reason in principle that we couldn't have the test runner automatically rerun tests with known intermittent failures a few times, and let the test

Re: Policy for disabling tests which run on TBPL

On 07/04/14 11:49 AM, Aryeh Gregor wrote: On Mon, Apr 7, 2014 at 6:12 PM, Ted Mielczarek wrote: If a bug is causing a test to fail intermittently, then that test loses value. It still has some value in that it can catch regressions that cause it to fail permanently, but we would not be able to

Re: Hi, I want to use mach in a project that use XUL SDK without building mozilla again and again.

On 24/06/14 09:42 PM, luoyongg...@gmail.com wrote: I want it can generate Microsoft Visual Studio Project and without the need to do configure things in msys. So that ease our development progress, and for our own purpose, I will create my own executable and besides, the unit testing/auto test

How many tests are disabled? (answer inside)

Something we've been missing for a long time is high level visibility into the state of our tests. How many tests from suite X are we skipping on platform Y? A new project called Test Informant [1] is keeping track of this data and allows us to generate reports with it. Here is a report genera

Re: How many tests are disabled? (answer inside)

On 30/09/14 11:19 AM, jmath...@mozilla.com wrote: Is there any chance you could generate a report on tests disabled in e10s mode? This would be very useful for the e10s team as we currently have a lot of tests disabled which need to get fixed. It should be do-able, I'll look into it. I think

Re: How many tests are disabled? (answer inside)

On 30/09/14 03:12 PM, Chris Peterson wrote: Is there a stable URL that always points to the most recent report? I recommend creating a symlink called something like "informant-report.html" or "latest.informant-report.html" to the most recent report. ("index.html" would be a bad name because it pr

Re: How many tests are disabled? (answer inside)

latest.informant-report.html symlink added! On 30/09/14 03:12 PM, Chris Peterson wrote: On 9/30/14 7:35 AM, Andrew Halberstadt wrote: A new project called Test Informant [1] is keeping track of this data and allows us to generate reports with it. Here is a report generated from yesterday&#

Test Informant Report - Week of 2014-09-28

See: http://people.mozilla.org/~ahalberstadt/informant-reports/weekly/2014-10-06.informant-report.html Hi all, last week I made a post saying to expect to see more of these reports. This is the first of a series of weekly reports I intend to post to dev.platform outlining which tests have been

Test Informant Report - Week of 2014-10-05

Test Informant report for 2014-10-12. State of test manifests at revision f547cf19d104. Using revision 9ee9e193fc48 as a baseline for comparisons. Showing tests enabled or disabled between 2014-10-06 and 2014-10-12. 79% of tests across all suites and configurations are enabled. Summary ---

Re: Test Informant Report - Week of 2014-10-05

On 14/10/14 11:58 AM, Bill McCloskey wrote: On Tue, Oct 14, 2014 at 8:45 AM, Dave Townsend wrote: Is there any way to see the breakdown of which testsgot enabled. I'm surprised to see mochitest-plain rise by 3456 yet mochitest-plain-e10s rise by only 28. I'd love to find out which tests got tur

  1   2   >