Re: Project Stockwell - February 2017 update

[Andrew Halberstadt]

Just noticed no one looped back here. Joel filed bug 1337844
 and bug 1337839
. There has been some
discussion there. To summarize, running tests locally is currently
optimized towards "Run all tests related to code in " instead of "Run
all tests in ". Optimizing for one by default, comes
with a trade off on the other.

That being said, I think there is some low hanging fruit that could make
the overall situation better, namely:

* Ability to run manifests in the args, e.g: ./mach mochitest
path/to/manifest.ini (this would bypass subdirs)
* An overall summary (bug 1209463
)
* A mode to prevent multi-Firefox instances from running (we would error
out if e.g multiple dirs or subsuites would be run)
* Advertising/bootstrapping aliases for common configurations, e.g add the
following to ~/.mozbuild/machrc:
[alias]
mochitest-media = mochitest -f plain --subsuite media

I agree that this kind of stuff is important, though can't make promises on
a timeline.
-Andrew


On Fri, Feb 10, 2017 at 10:24 AM, Mike Conley  wrote:

> There's good feedback in here. Are some of these known, jmaher? Are any
> intentional choices, or should we just start turning these into bugs to
> get fixed?
>
> On 08/02/2017 12:33 AM, Bill McCloskey wrote:
> > Hi Joel,
> > I spent about an hour tonight trying to debug a test failure, and I'm
> > writing this email in frustration at how difficult it is. It seems like
> the
> > process has actually gotten a lot worse over the last few years (although
> > it was never good). Here's the situation I ran into:
> >
> > A test is failing on try. I want to reproduce it. Assume that running the
> > test by itself isn't sufficient. I would like to run whatever set of
> tests
> > actually ran together on the test machine in a single Firefox
> invocation. I
> > don't want any more tests to run than those. I can't figure out any way
> to
> > do that.
> >
> > I can pass a directory to |mach mochitest|. But that has a number of
> > problems:
> > - It also runs every subdirectory recursively inside that directory.
> Often
> > that includes way more tests. I can't figure out any way to stop it from
> > doing this. I tried the "--chunk-by-dir" option, but it complains that
> the
> > argument is supposed to be an integer. What is this option for?
> > - |mach mochitest| runs all flavor of tests even though I only want one.
> > There is the --flavor option to disable that. But I have never figured
> out
> > how to use it correctly. No matter what I do, some irrelevant devtools
> are
> > a11y or plugin tests seem to run instead of what I want.
> > - There is a --start-at option that should allow me to start running
> tests
> > near the one that I want. But it never seems to work either. I'm not sure
> > if it's confounded by the two problems above, or if it's just completely
> > broken.
> >
> > We could easily fix this by printing in the tinderbox log the mach
> command
> > that you need to run in order to run the tests for a particular directory
> > (and making that discoverable through treeherder).
> >
> > I want to emphasize that, from a developer's perspective, this is the
> > second most basic thing that I could ask for. (Simply running a test by
> > itself is the most basic, and it works fine.) Running tests by directory
> in
> > automation has been a huge improvement, but we're not really earning the
> > dividends from it because it's so hard to get the same behavior locally.
> >
> > Anyway, sorry about the rant. And sorry to pick on your email. But it's
> > frustrating to see all these advanced ideas being proposed when we can't
> > even get basic stuff right.
> >
> > As an aside, I would also like to complain about the decision to strip a
> > lot of the useful information out of test logs. I understand this was
> done
> > to make the tests faster, and I can "just" check in a patch to add
> > "SimpleTest.requestCompleteLog()" to the intermittent test. But why
> didn't
> > we instead figure out why logging was so slow and fix that?
> Fundamentally,
> > it doesn't seem like saving 50MB of log data to S3 should take very long.
> >
> > -Bill
> >
> > On Tue, Feb 7, 2017 at 9:40 AM,  wrote:
> >
> >> This is the second update of project stockwell (first update:
> >> https://goo.gl/1X31t8).
> >>
> >> This month we will be recommending and asking that intermittent failures
> >> that occur >=30 times/week be resolved within 2 weeks of triaging them.
> >>
> >> Yesterday we had these stats:
> >> Orange Factor: 10.75 (https://goo.gl/qvFbeB)
> >> count(high_frequency_bugs): 61
> >>
> >> Last month we had these stats:
> >> Orange Factor: 13.76 (https://goo.gl/o5XOof)
> >> count(high_frequency_bugs): 42
> >>
> >> For more details of the bugs and what we are working on, you can read
> more
> >> on this recent blog post:
> >> https://elvis314.wordpress.com/2017/02/

Re: Sheriff Highlights and Summary in February 2017

[Andrew Halberstadt]

I don't have any data to back this up, but my suspicion is that a large
percentage
of backouts had try runs, but said try runs didn't run the jobs that failed
and caused
the backout. Imo, these kinds of backouts are (more) acceptable because it
means
the developer was trying to avoid doing a full try run, a practice that
should be
cheaper overall in the long run (and if done properly).

For example, you could either:
A) Do a full try run then push, almost guaranteeing you won't be backed
out. But
then you run every job twice and take longer to complete your bug, a
significant
cost.

B) Do a partial try run, running X% of the jobs yielding a Y% chance of
being
backed out.

There's some sweet spot between no try run, and try: -all which is the most
cost
effective (both in terms of AWS bill and developer time).

That being said, I think this is an issue of tools rather than policy.
Things like being
smarter about running jobs based on files changed and improving interfaces
to
selecting jobs on try, should help with backout rates. But the single
biggest thing we
could do is getting rid of integration branches altogether (and instead get
autoland to
merge directly from try). In this world, backouts would hardly even exist
anymore.

I believe we're already headed in this direction, albeit slowly.

-Andrew

On Fri, Mar 10, 2017 at 8:55 AM, David Burns  wrote:

> I went back and did some checks with autoland to servo and the results are
> negligible. So from 01 February 2017 to 10 March 2017 (as of sending this
> email). I have removed merge commits from the numbers.
>
> Autoland:
> Total Servo Sync Pushes: 152
> Total Pushes: 1823
> Total Backouts: 144
> Percentage of backouts: 7.8990674712
> Percentage of backouts without Servo: 8.61759425494
>
> Mozilla-Inbound:
> Total Pushes: 1472
> Total Backouts: 166
> Percentage of backouts: 11.277173913
>
>
> I will look into why, with more pushes, is resulting in fewer backouts. The
> thing to note is that autoland, by its nature, does not allow us to fail
> forward like inbound without having to get a sheriff to land the code.
>
> I think, and this is my next area to investigate, is the 1 bug per push
> (the autoland model) could be helping with the percentage of backouts being
> lower.
>
> David
>
> On 7 March 2017 at 21:29, Chris Peterson  wrote:
>
> > On 3/7/2017 3:38 AM, Joel Maher wrote:
> >
> >> One large difference I see between autoland and mozilla-inbound is that
> on
> >> autoland we have many single commits/push whereas mozilla-inbound it is
> >> fewer.  I see the Futurama data showing pushes and the sheriff report
> >> showing total commits.
> >>
> >
> > autoland also includes servo commits imported from GitHub that won't
> break
> > Gecko. (They might break the linux64-stylo builds, but they won't be
> backed
> > out for that.)
> >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Errors found by cppcheck

[Andrew Halberstadt]

I was doing a bit of research into cppcheck [1] to see if it might be worth
running as a linter (alongside eslint, flake8 etc). More discussion in bug
1370292 [2]. I ran it against the entire tree [3] and got these results:
https://bug1370292.bmoattachments.org/attachment.cgi?id=8874498

So far it looks like the false positive rate is too high to warrant
standing up a job in CI. It seems it would cause more frustration than it's
worth. But there are likely some legit errors in there, so please take a
quick look to see if any apply to your module. Feel free to comment over in
bug 1370292 if you have opinions one way or the other on standing this up
as a task. Barring a claim that it would be useful, I'll be WONTFIXing that
bug in a bit.

-Andrew


[1] http://cppcheck.sourceforge.net/
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1370292
[3] cppcheck $(sed -e 's/^/-i/' tools/rewriting/ThirdPartyPaths.txt)
-ithird_party . 2> errors.txt
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Errors found by cppcheck

[Andrew Halberstadt]

Yes, we can blacklist checks with:
cppcheck --suppress=

You can see the check ids with:
cppcheck --errorlist

It's actually even more flexible, checks can be disabled for specific
files, directories and even lines in the file. And it can all be specified
in a text file then passed in with:
cppcheck --suppressions-list=

Maybe we could start off running this as tier 3 (so it won't cause
backouts, but people who care can check up on it once in awhile). We could
modify the list of checks to exclude over time and maybe get a better sense
of whether or not the job is catching anything useful.

On Sun, Jun 18, 2017 at 3:08 PM, Ehsan Akhgari 
wrote:

> Is it possible to run cppcheck in a mode where we select which checks it
> runs?  If yes, we could look at the list of true positives that people have
> found (and continue to find) in bug 1370292 and create an opt-in set of
> checks that rare valuable ones to have turned on by default, and leave the
> rest off by default. (I'm hoping the effectiveness of its check isn't
> context sensitive...)
>
>
>
> On 06/06/2017 09:06 AM, Andrew Halberstadt wrote:
>
>> I was doing a bit of research into cppcheck [1] to see if it might be
>> worth
>> running as a linter (alongside eslint, flake8 etc). More discussion in bug
>> 1370292 [2]. I ran it against the entire tree [3] and got these results:
>> https://bug1370292.bmoattachments.org/attachment.cgi?id=8874498
>>
>> So far it looks like the false positive rate is too high to warrant
>> standing up a job in CI. It seems it would cause more frustration than
>> it's
>> worth. But there are likely some legit errors in there, so please take a
>> quick look to see if any apply to your module. Feel free to comment over
>> in
>> bug 1370292 if you have opinions one way or the other on standing this up
>> as a task. Barring a claim that it would be useful, I'll be WONTFIXing
>> that
>> bug in a bit.
>>
>> -Andrew
>>
>>
>> [1] http://cppcheck.sourceforge.net/
>> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1370292
>> [3] cppcheck $(sed -e 's/^/-i/' tools/rewriting/ThirdPartyPaths.txt)
>> -ithird_party . 2> errors.txt
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Linting for common causes of oranges in mochitests (need ideas)

[Andrew Halberstadt]

I'm looking into creating custom eslint rules that aim to avoid common
causes of oranges in tests. We have an MDN page

containing some of these already. Some of those patterns might be pretty
hard to catch with a linter, but others look do-able. So far, I'm thinking
of adding rules for:

   - No arbitrary setTimeouts
   - No synthesizeKey without waiting for focus
   - No deleting original tab
   - ???

These rules would only be configured to run on test files. I have a few
questions:

   1. Would this cause too much of a burden for legitimate uses of those
   patterns?
   2. Do any of the other patterns on that page look feasible to implement
   as linters?
   3. Are there additional things not listed on that page that we could
   lint for?

I'd love to find volunteer(s) who are more familiar with writing mochitests
than myself (and who also care about this issue) to help brainstorm and
provide more in-depth feedback. I want to strike a balance between
preventing oranges and not over-burdening test authors, but I'm not sure
what that balance looks like.

Thanks in advance for your input,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linting for common causes of oranges in mochitests (need ideas)

[Andrew Halberstadt]

Great idea, I'll add that to the list! Times actually are on that page too,
I just didn't know about performance.now() as an alternative.

Also, it was pointed out to me that in order to answer question 1) it would
help to know how to disable these rules. Eslint rules can be disabled at
the line or file level via comments. They can also be disabled at the
directory level in .eslintrc.js. So anyone wanting to legitimately use
these patterns would need to figure out the most appropriate way to disable
the corresponding rule.


On Thu, Jul 6, 2017 at 3:45 PM, Chris Peterson 
wrote:

>
>
> On 7/6/17 11:47 AM, Andrew Halberstadt wrote:
>
> - Are there additional things not listed on that page that we could lint
> for?
>
>
> Do we want to discourage tests from using Date (`new Date` or
> `Date.now()`) for measuring time? Dates are affected by time zones, DST,
> and clock skew issues jumping forward or backward. The performance.now()
> API doesn't have any of those problems, plus it uses high-resolution
> timestamps with 5 microsecond resolution instead of 1 millisecond. In bug
> 1372261, mconley is changing the tps Talos test to use 
> performance.timing.navigationStart
> + performance.now().
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linting for common causes of oranges in mochitests (need ideas)

[Andrew Halberstadt]

On Fri, Jul 7, 2017 at 12:15 PM, Ehsan Akhgari 
wrote:

> FWIW years ago I decided to act on trying to detect some of the patterns
> indicated on that page automatically, see https://bugzilla.mozilla.org/s
> how_bug.cgi?id=649012.  It didn't use linting (because we didn't really
> have any linting support back then) but the idea is the same, just with a
> different syntax.  That project never really received much (well, any!)
> traction since I always was hoping that people would triage the initial set
> of "opt-out" annotations that it landed with, and do something about those
> tests, but that never happened.  But hopefully this helped a bit by
> plugging one of the holes in terms of the unintended sources of
> intermittent oranges.
>
> In this case the way to opt out was adding a single line to the test.
> This wasn't hard per se, but if my memory serves it used to trip people for
> quite a while, even though the failure message you would get if you forgot
> to do it would indicate how to opt out of the extra check.  In hindsight, I
> think some people may have found the check more annoying than helpful,
> because in the cases where it helps, you'd get green tests, so nobody would
> consciously know that this is helping and in the cases where it would get
> in your way, people would be rightfully unhappy since their workflow got
> disturbed.
>

I forgot mochitest already checks for flaky timeouts! Maybe people would be
less opposed to a lint rule as many teams (especially frontend) have
already ingrained eslint into their workflows and know how to deal with
enabling/disabling rules. I guess in the setTimeout case we'd need to
choose one or the other, either eslint or requestFlakyTimeout. Doing both
at once would be a pain for developers. Fwiw, the eslint rule for
no-arbitrary-setTimeout turned out to be pretty easy to implement.



>
>>  1. Do any of the other patterns on that page look feasible to
>> implement as linters?
>>
>> I always struggled with this one.  Not many of them are, in my opinion.
> The example of using Date()s is, for example, but in my experience that is
> a super rare cause of intermittent failures...  A lot of these require some
> kind of dynamic check at runtime, and aren't easily detectible by just
> looking at the code of the test without running it.
>
>>
>>  1. Are there additional things not listed on that page that we could
>> lint for?
>>
>> Almost certainly.  Ideally when people fix an intermittent orange, they
> would think about whether this was a pattern that could reappear in other
> places, and whether it could be detected using a linter.  It is unclear to
> me how many of these new unique patterns we encounter though.  (For
> example, *many* intermittent oranges boil down to code expecting the wrong
> ordering of events, and that's not easily detectible using a linter
> But there may be a long tail of super rare error cases that are detectible
> using linters.)
>
> Hope this helps,
> Ehsan
>

Thanks for the insight. If we can block maybe 1 or 2 major causes of
intermittents with this, I'd be pretty happy. Maybe the bigger benefit of
this would be laying a foundation for adding new such linters in the
future, even if they only avoid minor causes of intermittents.

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

ESlint rule 'no-arbitrary-setTimeout' enabled on xpcshell tests

[Andrew Halberstadt]

As part of a larger effort to reduce oranges, we are starting to lint our
tests for common causes of intermittent failures. One low-hanging fruit is
preventing setTimeout with an arbitrary value (aka non-zero) as opposed to
waiting for an appropriate event. The mochitest harness already prevents
this in the harness itself (SimpleTest.requestFlakyTimeout), so this rule
is only enabled on xpcshell tests for now.

If you need to use a flaky setTimeout for some reason, you can disable the
rule at the directory level, file level or line level:
http://eslint.org/docs/user-guide/configuring#configuring-rules

It has been disabled in the following files due to pre-existing violations:
http://searchfox.org/mozilla-central/search?q=eslint-disable+mozilla%2Fno-arbitrary-setTimeout

Let me know if you think this should be enabled on any other test suites.
-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ESlint rule 'no-arbitrary-setTimeout' enabled on xpcshell tests

[Andrew Halberstadt]

Ah, good to know! I'll file a follow-up to enable the eslint rule on
browser/a11y/chrome. Maybe eventually we can just replace the
requestFlakyTimeout mechanism with this eslint rule. I decided to punt on
that as I'm not sure if eslint is running on 100% of mochitests yet.

On Fri, Jul 28, 2017 at 11:56 AM Felipe G  wrote:

> I'll note that requestFlakyTimeout is only enabled for mochitest-plain at
> the moment:
> http://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js#666
> So browser-chrome / a11y / chrome tests are still able to use non-0
> timeouts.
>
> Cheers,
> Felipe
>
> On Fri, Jul 28, 2017 at 12:48 PM, Andrew Halberstadt <
> ahalberst...@mozilla.com> wrote:
>
>> As part of a larger effort to reduce oranges, we are starting to lint our
>> tests for common causes of intermittent failures. One low-hanging fruit is
>> preventing setTimeout with an arbitrary value (aka non-zero) as opposed to
>> waiting for an appropriate event. The mochitest harness already prevents
>> this in the harness itself (SimpleTest.requestFlakyTimeout), so this rule
>> is only enabled on xpcshell tests for now.
>>
>> If you need to use a flaky setTimeout for some reason, you can disable the
>> rule at the directory level, file level or line level:
>> http://eslint.org/docs/user-guide/configuring#configuring-rules
>>
>> It has been disabled in the following files due to pre-existing
>> violations:
>>
>> http://searchfox.org/mozilla-central/search?q=eslint-disable+mozilla%2Fno-arbitrary-setTimeout
>>
>> Let me know if you think this should be enabled on any other test suites.
>> -Andrew
>>
> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

./mach try fuzzy: A Try Syntax Alternative

[Andrew Halberstadt]

I'm pleased to announce an alternative method for scheduling tasks on try
is now landed on mozilla-central. It makes use of the awesome fzf [1]
project to filter down the list of all task labels with a fuzzy matching
algorithm.

It works both with mercurial or git. If using mercurial, you'll need to
make sure you're updated to the latest version-control-tools:

$ ./mach mercurial-setup --update

To push to try, run:

$ ./mach try fuzzy

This will prompt you to install fzf. After bootstrapping is complete, it'll
generate the task list. This takes around ~10-20 seconds, but will be
cached so subsequent invocations won't incur this penalty again.

Now you'll be in the fzf interface. Basic usage is to start typing to
filter down the task list. You can use the arrow keys to move the cursor up
or down,  to select a task,  to select all tasks and 
to schedule the current selection (and their dependencies) on try.

There are a ton more keyboard shortcuts and features you can use to tweak
fzf just to your liking. For extra help, see:

$ ./mach try fuzzy --help
or
$ man fzf

For a demo and more information on implementation details, see:
https://ahal.ca/blog/2017/mach-try-fuzzy/

I expect this to work on all platforms including Windows for both mercurial
(with push-to-try) and git (with git-cinnabar). But it's a new tool without
a lot of real world testing, so a few bumps are likely. If you find any
bugs or bad UX, please file a bug under Testing :: General and let me know
(we should find a better component for this).

Cheers,
Andrew

p.s When running the fzf bootstrap, you'll be prompted to install some
shell extensions. These aren't necessary for |mach try fuzzy| to work so
feel free to hit 'N' at each prompt. That being said, the fzf shell
extensions are awesome. I highly encourage you to look into setting it up
for your shell. The fzf vim plugin is also great. See the project [1] for
more details.

[1] https://github.com/junegunn/fzf
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: reducing high try macosx pending counts

[Andrew Halberstadt]

Using the new |mach try fuzzy| selector will also make it a lot easier to
only schedule exactly what you need. To run what the above try syntax uses,
do:

$ ./mach try fuzzy
!osx 'web-platform 

That will run every task that doesn't contain the string 'osx', and does
contain the string 'web-platform'.

On Thu, Aug 3, 2017 at 3:51 AM James Graham  wrote:

> On 02/08/17 22:30, Kim Moir wrote:
> > You may have noticed that the time to wait for macosx test results on try
> > has been very long (>1day) this week.
> >
> > We have taken the following steps to address this problem
> [...]
>
> That sounds great! Thanks.
>
> For everyone else:
>
> It looks like the queues are still pretty long, and I imagine there are
> lots of pending jobs people scheduled that aren't really necessary any
> more (e.g. because you already found issues with a patch on other
> platforms, or landed in spite of the missing results).
>
> If you have a try push which requested OSX jobs, but you don't need them
> now, it would help if you go back and cancel the remaining jobs for that
> push from treeherder (look for the grey circle with a cross inside in
> the top right of the push to cancel all unfinished jobs, and the similar
> icon in the bottom panel for individual jobs). Also if you are making
> new try pushes and don't specifically need OSX testing, it's possible to
> limit tests to certain platforms with try syntax like (for running tests
> on linux64 and linux64-stylo only):
>
> try: -b do -p all -u web-platform-tests[x64,linux64-stylo]
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ESlint rule 'no-arbitrary-setTimeout' enabled on xpcshell tests

[Andrew Halberstadt]

This is now also enabled on browser-chrome tests. Bug 1389234 has been
filed to track deprecating SimpleTest.requestFlakyTimeout on
mochitest-plain and chrome in favour of this new rule.

-Andrew

On Fri, Jul 28, 2017 at 12:02 PM Andrew Halberstadt <
ahalberst...@mozilla.com> wrote:

> Ah, good to know! I'll file a follow-up to enable the eslint rule on
> browser/a11y/chrome. Maybe eventually we can just replace the
> requestFlakyTimeout mechanism with this eslint rule. I decided to punt on
> that as I'm not sure if eslint is running on 100% of mochitests yet.
>
> On Fri, Jul 28, 2017 at 11:56 AM Felipe G  wrote:
>
>> I'll note that requestFlakyTimeout is only enabled for mochitest-plain at
>> the moment:
>> http://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js#666
>> So browser-chrome / a11y / chrome tests are still able to use non-0
>> timeouts.
>>
>> Cheers,
>> Felipe
>>
>> On Fri, Jul 28, 2017 at 12:48 PM, Andrew Halberstadt <
>> ahalberst...@mozilla.com> wrote:
>>
>>> As part of a larger effort to reduce oranges, we are starting to lint our
>>> tests for common causes of intermittent failures. One low-hanging fruit
>>> is
>>> preventing setTimeout with an arbitrary value (aka non-zero) as opposed
>>> to
>>> waiting for an appropriate event. The mochitest harness already prevents
>>> this in the harness itself (SimpleTest.requestFlakyTimeout), so this rule
>>> is only enabled on xpcshell tests for now.
>>>
>>> If you need to use a flaky setTimeout for some reason, you can disable
>>> the
>>> rule at the directory level, file level or line level:
>>> http://eslint.org/docs/user-guide/configuring#configuring-rules
>>>
>>> It has been disabled in the following files due to pre-existing
>>> violations:
>>>
>>> http://searchfox.org/mozilla-central/search?q=eslint-disable+mozilla%2Fno-arbitrary-setTimeout
>>>
>>> Let me know if you think this should be enabled on any other test suites.
>>> -Andrew
>>>
>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>
>>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

PSA: Set prefs in mochitest manifests

[Andrew Halberstadt]

Bug 1328830  recently
landed and has added the ability to set prefs directly in a mochitest
manifest. Prefs can be set like this:

[DEFAULT]
prefs =
browser.newtabpage.introShown=true
layout.css.servo.enabled=true

[browser_foo.js]
[browser_bar.js]

There are a few limitations:

1. It must be set in the DEFAULT section. If you try to set prefs on an
individual test, it will error out.
2. Mochitest's --run-by-manifest mode must be enabled. This means that it
will work for all of 'plain', 'browser' and 'chrome'. It will not work for
'a11y' or 'jetpack*'.

The reason for these limitations is that we currently restart the browser
between each manifest when --run-by-manifest is enabled. So prefs can only
be set at the manifest level, and only if we restart Firefox beforehand.

I'm not aware of anyone using this feature yet, so if you run into bugs
please let me know!

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: disabled non-e10s tests on trunk

[Andrew Halberstadt]

We now have the ability to set prefs from a mochitest manifest (see bug
1328830  and my
recent newsgroup post). We could refactor these tests into a special
non-e10s manifest that sets browser.tabs.remote.autostart=false and keep
running them as part of the normal test run.

On Wed, Aug 9, 2017 at 5:28 PM Felipe G  wrote:

> I ran some scripts that I had to find out tests that are *fully* disabled
> on e10s, and posted the results to
> https://bugzilla.mozilla.org/show_bug.cgi?id=1376934
>
> In summary:
> mochitest-plain: 49 tests
> browser-chrome: 15 tests
> devtools: 86 tests
>
> Note that the script evaluates the skip-if condition for each test, so it's
> able to not count tests such as "skip=if e10s && debug" as fully disabled.
>
> On Wed, Aug 9, 2017 at 7:35 AM, Gabor Krizsanits 
> wrote:
>
> > On Wed, Aug 9, 2017 at 3:36 AM, Boris Zbarsky  wrote:
> >
> > >
> > > Hmm.  Do we load about:blank from the url bar in a content process?
> > >
> > >
> > Yes.
> >
> > I agree, I find it annoying too that we have to rely on
> > MOZ_DEBUG_CHILD_PROCESS
> > or MOZ_DEBUG_CHILD_PAUSE and that I have to be clever all the time about
> > how to hit the right process at the right time with the debugger. I never
> > switched back to non-e10s though since I don't trust that everything will
> > work the same and I don't think that should be the solution. Switching
> back
> > to single content process for debugging should come with less side
> effects
> > though... Also, this is not just an e10s/e10s-multi related issues we're
> > adding all kind of processes (extension/gpu/plugin/etc.).
> >
> > I didn't file a bug about this but I've been trying to find a decent
> > solution for it, but it seems like it's not trivial in any debugger
> (msvc,
> > gdb, lldb). Or maybe I was just hoping to find something better than what
> > seems to be achievable. Anyway, let's start with the bug: Bug 1388693.
> >
> > Gabor
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ./mach try fuzzy: A Try Syntax Alternative

[Andrew Halberstadt]

Btw, there's now a non-interactive mode landed with -q/--query. E.g to run
all browser-chrome:
./mach try fuzzy -q "'browser-chrome"

(You may want to first enter interactive mode to make sure your query will
actually select what you want, ctrl-c, then retype the query with -q so
it's in your shell history)

-Andrew

On Wed, Aug 2, 2017 at 10:30 AM Andrew Halberstadt 
wrote:

> I'm pleased to announce an alternative method for scheduling tasks on try
> is now landed on mozilla-central. It makes use of the awesome fzf [1]
> project to filter down the list of all task labels with a fuzzy matching
> algorithm.
>
> It works both with mercurial or git. If using mercurial, you'll need to
> make sure you're updated to the latest version-control-tools:
>
> $ ./mach mercurial-setup --update
>
> To push to try, run:
>
> $ ./mach try fuzzy
>
> This will prompt you to install fzf. After bootstrapping is complete,
> it'll generate the task list. This takes around ~10-20 seconds, but will be
> cached so subsequent invocations won't incur this penalty again.
>
> Now you'll be in the fzf interface. Basic usage is to start typing to
> filter down the task list. You can use the arrow keys to move the cursor up
> or down,  to select a task,  to select all tasks and 
> to schedule the current selection (and their dependencies) on try.
>
> There are a ton more keyboard shortcuts and features you can use to tweak
> fzf just to your liking. For extra help, see:
>
> $ ./mach try fuzzy --help
> or
> $ man fzf
>
> For a demo and more information on implementation details, see:
> https://ahal.ca/blog/2017/mach-try-fuzzy/
>
> I expect this to work on all platforms including Windows for both
> mercurial (with push-to-try) and git (with git-cinnabar). But it's a new
> tool without a lot of real world testing, so a few bumps are likely. If you
> find any bugs or bad UX, please file a bug under Testing :: General and let
> me know (we should find a better component for this).
>
> Cheers,
> Andrew
>
> p.s When running the fzf bootstrap, you'll be prompted to install some
> shell extensions. These aren't necessary for |mach try fuzzy| to work so
> feel free to hit 'N' at each prompt. That being said, the fzf shell
> extensions are awesome. I highly encourage you to look into setting it up
> for your shell. The fzf vim plugin is also great. See the project [1] for
> more details.
>
> [1] https://github.com/junegunn/fzf
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ./mach try fuzzy: A Try Syntax Alternative

[Andrew Halberstadt]

+to...@lists.mozilla.org

There have been a bunch of new features added here I'd like to highlight:

   - --artifact/--no-artifact: This works similarly to try syntax. If a
   local build with artifact builds is detected, --artifact will be used
   automatically. Unlike try syntax, this will change your treeherder symbols
   to 'Ba' to make it more obvious artifact builds are being used.
   - --env: You can now set environment variables in your tasks directly
   from the command line, e.g:
  - ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
  MOZ_LOG="example_logger:3"|
   - --full: Populate the fuzzy interface with the "full" set of tasks.
   This will let you choose tasks that were previously hard to select (e.g
   nightlies, l10n, signing tasks, etc)
   - --save/--preset: Works the same as try syntax, using the --query
   argument. E.g:
  - ./mach try fuzzy --save reftest -q "!cov !pgo 'reftest !jsreftest"
  - ./mach try --preset reftest

See |mach try fuzzy --help| for more information.

At this point I consider |mach try fuzzy| to be superior to try syntax in
almost every way [1], and would encourage people to start using it by
default. To do this, create a ~/.mozbuild/machrc file (if you don't already
have one) and add:
[try]
default = fuzzy

Now when you run |mach try| it will default to the fuzzy interface instead
of try syntax. If you need to use try syntax for some reason, you can still
run |mach try syntax -b do ... |. By the end of Q3 I plan on writing up an
outline to deprecate try syntax (it'll likely never be removed completely)
in favour of 'try_task_config.json' mechanisms. Nothing major will change
anytime soon, but it's worth being aware that there are now better methods
for scheduling tasks and finding people willing to maintain try syntax is
going to be even harder than it was before.

-Andrew

[1] Try syntax still has a few benefits over fuzzy. You can specify paths,
use --rebuild and it will detect compiled tests and error out if you try to
use artifact builds. There are bugs on file to reach parity here.

On Wed, Aug 2, 2017 at 10:30 AM Andrew Halberstadt 
wrote:

> I'm pleased to announce an alternative method for scheduling tasks on try
> is now landed on mozilla-central. It makes use of the awesome fzf [1]
> project to filter down the list of all task labels with a fuzzy matching
> algorithm.
>
> It works both with mercurial or git. If using mercurial, you'll need to
> make sure you're updated to the latest version-control-tools:
>
> $ ./mach mercurial-setup --update
>
> To push to try, run:
>
> $ ./mach try fuzzy
>
> This will prompt you to install fzf. After bootstrapping is complete,
> it'll generate the task list. This takes around ~10-20 seconds, but will be
> cached so subsequent invocations won't incur this penalty again.
>
> Now you'll be in the fzf interface. Basic usage is to start typing to
> filter down the task list. You can use the arrow keys to move the cursor up
> or down,  to select a task,  to select all tasks and 
> to schedule the current selection (and their dependencies) on try.
>
> There are a ton more keyboard shortcuts and features you can use to tweak
> fzf just to your liking. For extra help, see:
>
> $ ./mach try fuzzy --help
> or
> $ man fzf
>
> For a demo and more information on implementation details, see:
> https://ahal.ca/blog/2017/mach-try-fuzzy/
>
> I expect this to work on all platforms including Windows for both
> mercurial (with push-to-try) and git (with git-cinnabar). But it's a new
> tool without a lot of real world testing, so a few bumps are likely. If you
> find any bugs or bad UX, please file a bug under Testing :: General and let
> me know (we should find a better component for this).
>
> Cheers,
> Andrew
>
> p.s When running the fzf bootstrap, you'll be prompted to install some
> shell extensions. These aren't necessary for |mach try fuzzy| to work so
> feel free to hit 'N' at each prompt. That being said, the fzf shell
> extensions are awesome. I highly encourage you to look into setting it up
> for your shell. The fzf vim plugin is also great. See the project [1] for
> more details.
>
> [1] https://github.com/junegunn/fzf
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ./mach try fuzzy: A Try Syntax Alternative

[Andrew Halberstadt]

On Sat, Sep 2, 2017 at 1:00 PM Randell Jesup  wrote:

> >+to...@lists.mozilla.org
> >
> >There have been a bunch of new features added here I'd like to highlight:
>
> >   - --env: You can now set environment variables in your tasks directly
> >   from the command line, e.g:
> >  - ./mach try fuzzy --env XPCOM_DEBUG_BREAK=stack --env
> >  MOZ_LOG="example_logger:3"|
>
> This is *awesome*; I've been wanting this for YEARS.  Does this work
> without 'fuzzy'?
>

Yes and no :).

There has been a --setenv option to try syntax for a fair while now, but
I don't think it works with all tasks (likely just test tasks like
mochitest,
reftest, xpcshell, etc). I haven't tried this myself though, I'm not even
sure
if it's still working or not.

The |mach try fuzzy| implementation is built on top of a much more general
purpose mechanism that isn't (and won't be) available to try syntax. The
main
benefit of the |mach try fuzzy| implementation is that it will set the env
in
every task that gets scheduled no matter what kind/type.


>   - --save/--preset: Works the same as try syntax, using the --query
> >   argument. E.g:
> >  - ./mach try fuzzy --save reftest -q "!cov !pgo 'reftest !jsreftest"
> >  - ./mach try --preset reftest
>
> Also really great.
>
> --
> Randell Jesup, Mozilla Corp
> remove "news" for personal email
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intermittent oranges and when to disable the related test case - a simplified policy

[Andrew Halberstadt]

On Fri, Sep 8, 2017 at 7:10 PM Gregory Szorc  wrote:

> I know we've topic in this topic in the past but I can't recall outcomes.
> Is it worthwhile to define and use a richer test manifest "schema" that
> will facilitate querying and building dashboards so we have better
> visibility into disabled tests?
>

I don't think so, that data already exists and is query-able from
ActiveData:
https://activedata.allizom.org/tools/query.html#query_id=8pDOpeni

We just need to build a dashboard around something like that query. I had
previously written a pure JS tool that did this [1], but it was clunky and
poorly written. But someone with some actual frontend experience could
probably whip something up pretty fast.

Another thing we can (and should) do is write a mach command to trigger the
build system manifest parsing and return the results. This could possibly
be a mode on |mach test|.

[1] https://github.com/ahal/test-informant
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intermittent oranges and when to disable the related test case - a simplified policy

[Andrew Halberstadt]

On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan 
wrote:

> On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt <
> ahalberst...@mozilla.com> wrote:
>
>> I don't think so, that data already exists and is query-able from
>> ActiveData:
>> https://activedata.allizom.org/tools/query.html#query_id=8pDOpeni
>
>
> That query tells you about disabled tests, but doesn't know about *why* a
> test was disabled. E.g. you can't distinguish tests disabled because
> they're not expected to work on some (or all) platforms from tests that
> were disabled for intermittent failures that should, in principle, be fixed.
>
> Rob
>

True, though I don't know that gps' proposal would solve that either.

But this is a good idea, and is easy to solve from a technical standpoint.
We'd just need to agree on some standard manifest keys:

[test_foo.html]
skip-if = 
reason = {'intermittent', 'fail', ... }
bugs = 1234567, 1234576

When we log the skip, we make sure this metadata makes it into the
structured log. Then tools like ActiveData or a manifest parsing mach
command can query it. Reftest and wpt complicate matters a bit, but we
could figure out something similar for them.

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reminder on Try usage and infrastructure resources

[Andrew Halberstadt]

There's sort of a way to do this with try syntax. I say sort of because it
doesn't support all suites and there seems to be a few bugs with it. But
you can try:

./mach try -b o -p linux64 -u none path/to/dir/or/test

This should only run the directory or test you specified (it'll always show
up as chunk 1). I have vague plans to implement this a bit more robustly
for try_task_config.json based scheduling, but no time frame on when that
work might happen yet.

-Andrew

On Thu, Sep 14, 2017 at 11:48 AM Marco Bonardo  wrote:

> When I need to retrigger a mochitest-browser test multiple times (to
> investigate an intermittent), often I end up running all the
> mochitest-browser tests, looking at every log until I find the chunk
> where the test is, and retrigger just that chunk. The chunk number
> changes based on the platform and debug/opt, so it's painful.
> Is there a way to trigger only the chunk that will contain a given
> test, so I can save running all of the other chunks?
>
> --
> You received this message because you are subscribed to the Google Groups
> "firefox-ci" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to firefox-ci+unsubscr...@mozilla.com.
> To post to this group, send email to firefox...@mozilla.com.
> To view this discussion on the web visit
> https://groups.google.com/a/mozilla.com/d/msgid/firefox-ci/CAPDqYT151ETZSGM83Wo_jdpSj1bHhs57eTpah4bE5PE2BM9ckQ%40mail.gmail.com
> .
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reminder on Try usage and infrastructure resources

[Andrew Halberstadt]

Yes, all mochitests except Android restart between manifests (which is
usually the same as folders).

On Thu, Sep 14, 2017 at 12:03 PM Marco Bonardo  wrote:

> On Thu, Sep 14, 2017 at 5:56 PM, James Graham 
> wrote:
> > On 14/09/17 16:48, Marco Bonardo wrote:
> > mach try -p linux64 
>
> Afaict, that runs a single folder, but the intermittent may be caused
> by interactions across different tests in different folders. I'm not
> up-to-date to what we do today, do we restart the test harness per
> each folder? That'd basically solve my troubles.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to require `mach try` for submitting to Try

[Andrew Halberstadt]

Yes, we'd like to make it the default eventually. I've been holding off for
now, but expect it will be switched to the default at some point in Q4 or
Q1. If you don't want to wait and are tired of typing 'fuzzy' each time,
you can create a ~/.mozbuild/machrc file and add:
[try]
default = fuzzy

On Fri, Sep 15, 2017 at 7:54 PM Bobby Holley  wrote:

> FWIW, I've found |./mach try fuzzy| to be very intuitive and kind of the
> best of both worlds for command-line vs GUI. I don't know what the
> non-fuzzy version does, but perhaps fuzzy should be the default.
>
> On Fri, Sep 15, 2017 at 4:51 PM, Nicholas Nethercote <
> n.netherc...@gmail.com
> > wrote:
>
> > Are there docs on how to use `./mach try`? `./mach help try` doesn't give
> > much detail.
> >
> > Also, will https://mozilla-releng.net/trychooser/ still work? I'm
> > generally
> > more of a command line person than a GUI person, but this is one case
> where
> > I find the GUI approach far more usable.
> >
> > Nick
> >
> > On Sat, Sep 16, 2017 at 8:30 AM, Gregory Szorc  wrote:
> >
> > > The Try Service ("Try") is a mechanism that allows developers to
> schedule
> > > tasks in automation. The main API for that service is "Try Syntax"
> (e.g.
> > > "try: -b o -p linux -u xpcshell"). And the transport channel for making
> > > that API call is a Mercurial changeset's commit message plus a version
> > > control "push" to the "try" repository on hg.mozilla.org.
> > >
> > > As the recent "Reminder on Try usage and infrastructure resources"
> thread
> > > details, scaling Firefox automation - and Try in particular - is
> > > challenging. In addition, the number of workflows and variations that
> > > automation needs to support is ever increasing and continuously
> evolving.
> > >
> > > It shouldn't come as a surprise when I say that we've outgrown many of
> > the
> > > implementation details of the Try Service. Try Syntax itself is over 7
> > > years old and has survived a complete rewrite of automation scheduling,
> > for
> > > example. Aspects of Try are adversely impacting the ability for
> > developers
> > > to use Try efficiently and therefore undermining our collective ability
> > to
> > > get important work done quickly.
> > >
> > > In order to put ourselves in a position to more easily change
> > > implementation details of the Try Service so we may deliver a better
> > > experience for all, we'd like to require the use of `mach try` for Try
> > > submissions. This will ensure there is a single, well-defined, and
> > > easily-controlled mechanism for submitting to Try. This will allow
> > greater
> > > flexibility and adaptability. It will provide better opportunities for
> > user
> > > messaging. It will ensure that any new features are seen by everyone
> > > sooner. It will eventually lead to faster results on Try for everyone.
> > >
> > > Bug 1400330 ("require-mach-try") is on file to track requiring `mach
> try`
> > > to submit to Try.
> > >
> > > The mechanism for submitting to Try has remaining relatively stable for
> > > years. `mach try` is relatively new - and I suspect unused by a
> sizeable
> > > population. This is a potentially highly disruptive transition. That's
> > why
> > > we're not making it immediately and why we're sending this email today.
> > >
> > > You can mitigate the disruption by using `mach try` before the forced
> > > transition is made and reporting bugs as necessary. Have them block
> > > "require-mach-try" if you need them addressed before the transition or
> > > "mach-try" otherwise. We don't really have a good component for `mach
> > try`
> > > bugs, so put them in TaskCluster :: Task Configuration for now and
> chain
> > > them to a tracking bug for visibility.
> > >
> > > FAQ
> > >
> > > Q: When will the transition be made?
> > > A: When we feel `mach try` is usable for important workflows (as judged
> > by
> > > blockers on "require-mach-try"). Also, probably not before Firefox 57
> > ships
> > > because we don't want to interfere with that release.
> > >
> > > Q: What about old changesets?
> > > A: You will still be able to submit to Try using the current/legacy
> > > mechanism for old changesets. There will be a "flag day" of sorts on
> > > mozilla-central after which all Try submissions will require `mach try`
> > or
> > > nothing will get scheduled.
> > >
> > > Q: Will Try Syntax continue to work?
> > > A: For the foreseeable future, yes. There is a long-term goal to
> replace
> > > Try Syntax with something more automatic and less effort - at least for
> > > most use cases. But there are no definite plans or a timetable to
> remove
> > > Try Syntax.
> > >
> > > Q: Are there any other major changes planned?
> > > A: Several. People are hacking on path-based selection, `mach try
> fuzzy`
> > > improvements, moz.build-based annotations influencing what gets
> > scheduled,
> > > not using a traditional Mercurial repository for backing Try, and more.
> > > Some of these may not be available to legacy Try submission wo

Re: Intent to require `mach try` for submitting to Try

[Andrew Halberstadt]

Interesting, please file a bug either way and CC me. It worked
for me back when it first landed, though I haven't tried it since.
I'll see if I can reproduce sometime this week.

On Sat, Sep 16, 2017 at 4:31 AM Marco Bonardo  wrote:

> I have a problem with try fuzzy, not sure if it's just my system, so
> didn't file a bug yet. On Windows it always hangs after it tries to
> cleanup its own stuff at the end, I must terminate it and manually
> uncommit and purge the config file.
> Does anyone have the same problem?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: --verify option added to mochitest, reftest, xpcshell test harnesses

[Andrew Halberstadt]

This is really great Geoff! Hopefully it can cut down the number of new
intermittents we introduce to the tree. Do you know if orangefactor or
ActiveData can track the rate of new incoming intermittents? Would be neat
to see how much of an impact this tool has on that.

On Mon, Oct 2, 2017 at 1:08 PM Geoffrey Brown  wrote:

> The mochitest, reftest, and xpcshell test harnesses now support a
> --verify option. For example:
>
>   mach mochitest
> docshell/test/test_anchor_scroll_after_document_open.html --verify
>
> In verify mode, the requested test is run multiple times, in various
> "modes", in hopes of quickly finding any intermittent failures. Once
> tests are complete, a summary of results is printed:
>
> :::
> ::: Test verification summary for:
> :::
> ::: docshell/test/test_anchor_scroll_after_document_open.html
> :::
> ::: 1. Run each test 10 times in one browser. : Pass
> ::: 2. Run each test 5 times in a new browser each time. : Pass
> ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
> ::: 4. Run each test 5 times in a new browser each time, in chaos mode. :
> Pass
> :::
> ::: Test verification PASSED
> :::
>
> There's no flexibility in the number of times the test is run in each
> mode and that's by design: I wanted a simple, standard way of checking
> "Is this test likely to produce a frequent intermittent failure"?
>
> Verify mode was developed for the test-verify task (announcement
> coming soon!) but it may also be a convenient addition to your local
> testing.
>
> More info at [1]. Bug and enhancement requests welcomed: please file
> bugs blocking bug 1357513.
>
> [1] https://developer.mozilla.org/en-US/docs/Test_Verification
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to Enable: Automated Static Analysis feedback in MozReview

[Andrew Halberstadt]

On Tue, Oct 17, 2017 at 4:00 PM  wrote:

> Those things can be encoded as regexps, but they span across programming
> languages (XUL, XHTML, HTML, XBL, DTD, JS, C++).
>

To create a new regex-based linter, simply add a file like this:
https://dxr.mozilla.org/mozilla-central/source/tools/lint/test-disable.yml

It'll get picked up automatically and be enabled on any files that match
the include/exclude directives. Just run:
./mach lint 

And anything that matches the regex will spit out an error. The above
link is currently the only regex based linter in-tree, so it could
definitely use some improvements (e.g we could allow a list of regexes
in a single linter). It also only works if the offending string is all on
the
same line, which isn't ideal.

Is there any mozilla-clippy project being planned? :)
>

Yes, see bug 1361342. We were blocking on rust stable support in clippy.
Last I heard that still hadn't landed (though this was several months
ago). If anyone knows the status there please chime in on the bug.


> zb.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to require Python 3 to build Firefox 59 and later

[Andrew Halberstadt]

On Fri, Nov 10, 2017 at 9:44 PM David Burns  wrote:

> My only concern about this is how local developer environments are going
> to be when it comes to testing. While I am sympathetic to moving to python
> 3 we need to make sure that all the test harnesses have been moved over and
> this is something that needs a bit of coordination. Luckily a lot of the
> mozbase stuff is already moving to python 3 support but that still means we
> need to have web servers and the actual test runners moved over too.
>
> David
>

For libraries like mozbase, I think the plan will be to support both 2 and
3 at
the same time. There are libraries (like 'six') that make this possible.
I'd bet
there are even parts of the build system that will still need to support
both at
the same time.

With that in mind, I don't think python 3 support for test harnesses needs
to
block the build system.

Andrew

On 10 November 2017 at 23:27, Gregory Szorc  wrote:
>
>> For reasons outlined at
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1388447#c7, we would like
>> to make Python 3 a requirement to build Firefox sometime in the Firefox 59
>> development cycle. (Firefox 59 will be an ESR release.)
>>
>> The requirement will likely be Python 3.5+. Although I would love to make
>> that 3.6 if possible so we can fully harness modern features and
>> performance.
>>
>> I would love to hear feedback - positive or negative - from downstream
>> packagers and users of various operating systems and distributions about
>> this proposal.
>>
>> Please send comments to dev-bui...@lists.mozilla.org or leave them on
>> bug 1388447.
>>
>> ___
>> dev-builds mailing list
>> dev-bui...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-builds
>>
>> ___
> dev-builds mailing list
> dev-bui...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: overly strict eslint rules

[Andrew Halberstadt]

Not replying to anyone specifically, but Sylvestre's team is working on
getting our linters hooked up to mozreview/phabricator as well. I think
this paired with bootstrapping the hooks will smooth out the lint fixing
workflow considerably.

Andrew


On Fri, Jan 5, 2018 at 5:58 AM Mark Banner  wrote:

> On 03/01/2018 14:57, Mark Banner wrote:
> > On 24/12/2017 22:07, Masatoshi Kimura wrote:
> >> ---
> >> $ mach lint browser/base/content/aboutDialog.js
> >> eslint-plugin-html v2.0.3 should be v4.0.0.
> >> ESLint is an old version, clobbering node_modules directory
> >> Clobbering node_modules...
> >> Installing eslint for mach using
> >> "d:\mozilla-build\node-v8.1.4-win-x64\npm.cmd install
> >> --loglevel=error"...
> >> npm ERR! code ENOLOCAL
> >> npm ERR! Could not install from
> >> "tools\lint\eslint\eslint-plugin-mozilla" as it does not contain a
> >> package.json file.
> > In this case, I think it incorrectly removed the
> > tools\lint\eslint\eslint-plugin-mozilla directory, if you check your
> > source tree diffs, you should see if that was the case or not (though
> > since this was a while ago, you may have already found that by now).
> >
> > I've a feeling I know why, unfortunately my windows environment isn't
> > very good at the moment, so I'll need to get that updated to investigate.
> Just to follow up on this, this was a one-off issue due to a bad clobber
> when we attempted to update from v3 to v4 of ESLint. I've tracked down
> the issue , and
> I'm near to a patch.
>
> The workaround is running a revert on tools/lint/eslint/ and then
> running eslint again - though I suspect as we did the upgrade a while
> ago, most people will have already fixed this themselves.
>
> Mark.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

PSA: Default log format changed for testing commands

[Andrew Halberstadt]

You should notice a new colourized log format when
running tests with |mach test| or |mach mochitest|.
If you want to revert to the old log format you can add:

[test]
format=tbpl

to your ~/.mozbuild/machrc. If you have any feedback
or issues with the new format, please chime in to
bug 1445624 .

A few notes:
- There are a few reports of colours not working on Windows (I'm
investigating)
- |mach reftest| still uses the old format for compatibility with the
reftest-analyzer

Thanks,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Removing tinderbox-builds from archive.mozilla.org

[Andrew Halberstadt]

Going back to the original question, it looks like mozregression doesn't
use the builds that Nick wants to remove anyway. So regardless of our
retention policies, it looks like removing these builds would have no
impact on mozregression's effectiveness. Is that an accurate statement?

-Andrew

On Wed, May 9, 2018 at 2:10 PM William Lachance 
wrote:

> On 2018-05-09 2:01 PM, Ted Mielczarek wrote:
> >> It's useful for tracking down regressions no matter how old the
> >> regression is; I pretty regularly see mozregression finding useful
> >> data on bugs that regressed multiple years ago.
> > To be clear here--we still have an archive of nightly builds dating back
> to 2004, so you should be able to bisect to a single day using that. We
> haven't ever had a great policy for retaining individual CI builds like
> these tinderbox-builds. They're definitely useful, and storage is not that
> expensive, but given the number of build configurations we produce nowadays
> and the volume of changes being pushed we can't archive everything forever.
>
> jrmuizel pointed this bug to me, I assume this kind of scenario is not
> that uncommon, where a manual/source bisection (which would presumably
> take a bunch of scarce developer time) is required due to the
> integration builds being expired:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1444055
>
> Based on the feedback I'm seeing here, I think it might be worth
> extending the expiry dates a little further back for some common build
> configurations used by mozregression. Obviously no one cares about a
> linux32 asan build from 2013, but I feel like there might be a better
> sweet spot than where we're at.
>
> Will
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

PSA: Setting preferences and extensions in test harnesses

[Andrew Halberstadt]

tl;dr - You can now set prefs and install extensions across multiple
harnesses by modifying the relevant profile under testing/profiles.

If you previously would have set a  pref in:
testing/profiles/prefs_general.js,

Use this instead:
testing/profiles/unittest/user.js


# Overview

I'm currently in the process of consolidating prefs and extensions
across all our test harnesses into a single shared location
(testing/profiles
). If you
navigate there you'll find several different
"profile-like" directories.

Each profile directory has a user.js file for setting prefs, and an
extensions dir for dropping in extensions. I call these directories
"profile-like" because they only currently support prefs and
extensions. Don't expect to be able to add other profile-related
files (though if there's a need, support for other kinds of profile
data can be implemented).

You'll also find a 'profiles.json' file, and a 'profile' utility script. The
JSON file is used to map test suites to a list of profile directories to
apply. For example, mochitest uses the 'common' and 'unittest'
profiles in that order. Prefs from the latter profiles will overwrite
prefs from the earlier ones. This means you can either set prefs for
a single harness (e.g by adding them to the latter profile), or set
prefs across all harnesses at the same time (e.g by adding them
to the 'common' profile). Same goes for extensions.

Because prefs are split across multiple directories, and those
directories can overwrite one another, the 'profile' utility script can
help you view and compare the contents of a profile. For example:

$ cd testing/profiles
$ ./profile show mochitest  # dumps all prefs that mochitest will use

You can also diff profiles or sort preference files alphabetically:

$ ./profile diff mochitest talos
$ ./profile sort mochitest

In the future, I may add a command to automatically set preferences
in a given list of suites.


# Motivations

One great use case is testing the impact prefs and extensions have
on our performance (and unittests). Simply drop an extension into
testing/profiles/common/extensions and push to try. It will be
installed in all of our harnesses (that use this system), including Talos.

Another reason for this change is that it provides more visibility into
which prefs are set in which harnesses. We can now diff the prefs set
in various suites to try and reason about whether those differences are
intentional or omissions.

Lastly, this will make it easier to set prefs. No more hunting around
the tree looking for preference files. This will help prevent cases
where a pref was accidentally omitted from a test harness.


# Further Work

I'm in the middle of migrating harnesses over to this new system.
Notably, reftest and xpcshell have not yet been converted, though
hopefully these will be done soon. There is a long tail of smaller
harnesses that I might not have time to get around to however.

For current support, just look here:
https://searchfox.org/mozilla-central/source/testing/profiles/profiles.json

Here's the relevant bug tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1451159&hide_resolved=0

Let me know if you have any questions or concerns,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Setting preferences and extensions in test harnesses

[Andrew Halberstadt]

On Wed, May 16, 2018 at 11:07 AM Andreas Tolfsen  wrote:

> Any plans to consolidate the Mn preferences, currently stored in
> geckoinstance.py?
>

Yes, but I don't have a timeline. I want to at least finish up reftest and
xpcshell first. Then time permitting, I'd like to finish up marionette,
cppunittest, etc, however this won't be a primary focus.

I just filed https://bugzilla.mozilla.org/show_bug.cgi?id=1462007 (feel
free to grab it if you'd like to see this sooner).
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: OrangeFactor/Intermittent Failures View Update

[Andrew Halberstadt]

Thanks for all your work here, this looks much cleaner and easier to
interpret. Looking forward to the future improvements you mentioned!

For anyone bad at remembering URLs, you can get to this view by clicking
the "Treeherder" logo in the top left of the main treeherder page.

-Andrew

On Wed, Jun 6, 2018 at 8:34 AM Andrew Halberstadt  wrote:

> Thanks for all your work here, this looks much cleaner and easier to
> interpret. Looking forward to the future improvements you mentioned!
>
> For anyone bad at remembering URLs, you can get to this view by clicking
> the "Treeherder" logo in the top left of the main treeherder page.
>
> -Andrew
>
> On Tue, Jun 5, 2018 at 7:23 PM Robert O'Callahan 
> wrote:
>
>> Thank you very very much for making this information public again!
>>
>> Rob
>> --
>> Su ot deraeppa sah dna Rehtaf eht htiw saw hcihw, efil lanrete eht uoy ot
>> mialcorp ew dna, ti ot yfitset dna ti nees evah ew; deraeppa efil eht.
>> Efil
>> fo Drow eht gninrecnoc mialcorp ew siht - dehcuot evah sdnah ruo dna ta
>> dekool evah ew hcihw, seye ruo htiw nees evah ew hcihw, draeh evah ew
>> hcihw, gninnigeb eht morf saw hcihw taht.
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: pay attention when setting multiple reviewers in Phabricator

[Andrew Halberstadt]

It might be worth investigating whether we can switch Phabricator's
default (so that multiple reviews are all blocking, and to make them
non-blocking would require the extra step). Personally I think setting
multiple reviewers up on a first come first serve is disrespectful to
those reviewers' time, so shouldn't be the case that gets optimized.

Though I also like that the distinction can be made, and there are cases
where first-come-first-serve would be a good idea (e.g things that need
to land ASAP).

On Tue, Jul 3, 2018 at 1:29 AM glob  wrote:

> Jean-Yves Avenard wrote on 3/7/18 6:23 am:
>
> On Mon, Jul 2, 2018 at 5:01 PM, Andreas Tolfsen  wrote:
>
>> Also sprach Marco Bonardo:
>>
>> > When asking for review to multiple reviewers, and all of them must
>> accept
>> > your revision, you must mark them as blocking reviews, either in the
>> > Phabricator ui or appending "!" at the end of the reviewer name.
>> Otherwise
>> > it's first-come-first-serve.
>>
>> Note that is and also has been the case for mozreview.
>>
>
> I don't ever recall mozreview having different kind of reviewer (blocker
> or non-blocker), if two people were added as reviewer, by default both had
> to review.
>
> it's correct that mozreview (and bugzilla) only have one type of
> reviewer.  what multiple reviewers means in bugzilla/mozreview varies from
> team to team (all must review vs. any can review).
>
> it isn't correct that in mozreview two reviewers would both have to review.
> approval from _any_ reviewer would allow it to be landed with autoland:
>
> https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/pylib/mozreview/mozreview/review_helpers.py#l34
>
> i like that phabricator makes this distinction up-front.
> thanks mak for drawing attention to this difference/feature.
>
>
>
> -glob
> --
> glob — engineering workflow — moz://a
>
> ___
> firefox-dev mailing list
> firefox-...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Automated code analysis now also in Phabricator

[Andrew Halberstadt]

Congrats, thanks to everyone involved for getting this working!

I really like the idea of comparing errors with and without the patch,
this would let us lint code where linting isn't explicitly enabled in
mach lint/CI. One caveat to doing is that the review bot would need
to make it very clear which issues are "optional" (aka can safely be
ignored) and which ones are "required" (aka would cause a backout
if they don't get fixed).

I have some ideas of making this distinction directly in mozlint by
treating all "warnings" as optional fixes and all "errors" as required
fixes. See https://bugzilla.mozilla.org/show_bug.cgi?id=1460856

Andrew

On Tue, Jul 17, 2018 at 9:22 AM Jan Keromnes  wrote:

> TL;DR -- “reviewbot” is now enabled in Phabricator. It reports potential
> defects in pending patches for Firefox.
>
> Last year, we announced Code Review Bot (“reviewbot”, née “clangbot”), a
> Taskcluster bot that analyzes every patch submitted to MozReview, in order
> to automatically detect and report code defects *before* they land in
> Nightly:
>
>
> https://groups.google.com/d/msg/mozilla.dev.platform/TFfjCRdGz_E/8leqTqvBCAAJ
>
> Developer feedback has been very positive, and the bot has caught many
> defects, thus improving the quality of Firefox.
>
> Today, we’re happy to announce that reviewbot analyzes every patch
> submitted to Phabricator as well.
>
> Here is an example of an automated review in Phabricator:
> https://phabricator.services.mozilla.com/D2120
>
> Additionally, we’ve made a number of improvements to this bot over the
> past months. Notably:
> - Enabled several clang-tidy checks to catch more C/C++ defects (e.g.
> performance and security issues)
> - Integrated Mozlint in order to catch JS/Python/wpt defects as well
> - Fixed several bugs, like the lowercase code snippets in comments
> - We’re now detecting up to 5x more defects in some patches
>
> Please report any bugs with the bot here: https://bit.ly/2tb8Qk3
>
> As for next steps, we’re currently discussing a few ideas for the
> project’s future, including:
> - Catch more bugs by comparing defects before & after a patch is applied
> (currently, we report defects located on lines that are directly modified
> by the patch)
> - Evaluate which defect types are generally being fixed or ignored
> - Evaluate analyzing Rust code with rust-clippy
> - Help with coding style by leveraging clang-format
> - Integrate more deeply with Phabricator, e.g. by reporting a build status
> for our analysis
> - Integrate our analysis with Try, in order to unify our various CI and
> code analysis tools
>
> Many thanks to everyone who helped make this a reality: Bastien, who did
> most of the implementation and bug fixing, Marco, Andi, Calixte, Sylvestre,
> Ahal, the Release Management Analysis team and the Engineering Workflow
> team.
>
> Jan
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

PSA: Re-run old (non-syntax) try pushes with |mach try again|

[Andrew Halberstadt]

While |mach try fuzzy| is generally a better experience than try
syntax, there are a few cases where it can be annoying. One
common case was when you selected a bunch of tasks in the
interface and pushed. Then at a later date you wanted to push
the exact same set of tasks again. This used to be a really poor
experience as you needed to re-select all the same tasks
manually.

As of now, you can use |mach try again| instead. The general
workflow is:

 $ ./mach try fuzzy

...


$ ./mach try again


*More Details*

Whenever you push to try with a `try_task_config.json` (aka
anything but try syntax), that config will be saved in a history
file (in ~/.mozbuild/history/try_task_configs.json). You can view
your history keyed by commit message by running:

$ ./mach try again --list

If you wish to re-push an older `try_task_config.json`, you can
use:

$ ./mach try again --index 

Where  is the number displayed by --list. By default the
last 10 pushes will be saved in history, but this can be
configured by setting:

[try]
maxhistory=

in your ~/.mozbuild/machrc.


*Caveats*

As mentioned earlier, pushes with try syntax aren't saved in
history as they don't use `try_task_config.json` files. When/if
bug 1400295 lands, then pushes generated with try syntax
will start working.

The full taskgraph might change between the time you
originally ran the push and the time you re-push with |mach try
again|. If this happens, it's possible the tasks you previously
scheduled were renamed (or removed) and the decision task
will fail.

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ./mach try fuzzy: A Try Syntax Alternative

[Andrew Halberstadt]

I recently added the ability to specify --query multiple
times (where the set of tasks is the union of each
individual query). So something like:

./mach try fuzzy -q "'android !pgo !cov" -q "'build !pgo !cov"

Should also accomplish what you want. It's still a bit clunky
as multiple queries don't work with --save yet (just the first
query will be saved). But at least this will get the command
in your shell history (and you can also use |mach try again| to
rerun it).

On Mon, Aug 6, 2018 at 3:06 AM James Graham  wrote:

> On 06/08/2018 01:25, Botond Ballo wrote:
> > Is there an easy way to do a T-push (builds on all platforms, tests on
> > one platform only) with |mach try fuzzy|?
> >
> > I usually do T-pushes using try syntax, by Trychooser seems to be out
> > of date when it comes to building a T-push syntax for Android, so I'm
> > at a loss as to how to do a T-push for Android right now.
>
> There are a couple of options. Interactively you can select all the
> builds you want, press ctrl+a (or whatever the select-all keybinding you
> have configured is), then do the same again with the tests you want,
> then accept all your choices.
>
> If you want to construct a single query string that can be reused with
> --save, something like 'test-linux64 | build !ccov !pgo !msvc' seems to
> select all builds and tests just on linux64. Unfortunately I can't
> figure out any way to logically group expressions, which does make
> composing multiple terms more tricky.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Local static analysis builds on Linux and Mac OS X

[Andrew Halberstadt]

On 15/10/15 01:59 PM, Ehsan Akhgari wrote:

On 2015-10-14 9:18 PM, Mike Hommey wrote:

BTW, since we're growing the set of checks the plugin does, what do we
do to ensure that it doesn't add too much overhead to the compilation
time?


Currently nothing beyond taking care to not add slow checks.

Since we're talking about the build speed here, I hope that some day we
start tracking our build speed to understand how it regresses over time.
  Once we have that infrastructure, we can easily track this for static
analysis builds in addition to other kinds of builds.


This data is available in active data right now. Here's a really dumb 
query that returns a tuple of builder name and seconds it took the 
"mozharness build step" to complete:

http://activedata.allizom.org/tools/query.html#query_id=wIv23AVn

Someone who is better at querying than me could probably come up with 
something that returns a daily average by platform. See the "jobs" 
schema linked from the query tool for other parameters.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Merging comm-central into mozilla-central

[Andrew Halberstadt]

On 23/10/15 04:43 AM, Gregory Szorc wrote:


- It adds burden to developers, needing to support those projects
   merged from comm-central.
 Just look around in mozilla-central at all the optional things
 that are not visible on treeherder and break regularly. The
 situation wouldn't be different in that sense. But the people
 that do care about those projects will have a better experience
 supporting them.


The other serious concern is impact to automation. Are we going to close
trees for Thunderbird failures? Are we going to run Thunderbird jobs on
every push? (Do we do this now?)


This is a decision we can make independently of merging comm-central to 
mozilla-central. We have a lot of code living in mozilla-central that 
does not get tested per push, nor does it close the trees when it fails. 
Moving to m-c does not mean that the current state of affairs for 
thunderbird needs to change. Whether or not we'd like the current state 
of affairs for thunderbird to change is another matter, but it doesn't 
need to be a blocking issue w.r.t the merge.


Big +1 for the proposal btw.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Now measuring Firefox size per-commit. What else should we be tracking?

[Andrew Halberstadt]

ActiveData is just a very large data base. An automated-client would be
something that periodically runs a query, formats the data and plugs it
into a graph. Here's an example of a client side JS tool that runs a
query to determine which tests are enabled or skipped:
https://github.com/mozilla/test-informant/blob/master/js/report.js#L105

But I think trying to track runtime at the job (aka chunk) level is
going to be way too noisy. Something more useful might be (total suite
runtime)/(total number of tests) across all chunks.

-Andrew

On 05/11/15 04:18 AM, L. David Baron wrote:

On Wednesday 2015-11-04 12:46 -0500, William Lachance wrote:

On 2015-11-04 10:55 AM, William Lachance wrote:


1. Relatively deterministic. 2. Something people actually care
about and are willing to act on, on a per-commit basis. If you're
only going to look at it once a quarter or so, it doesn't need to
be in Perfherder.

Anyway, just thought I'd open the floor to brainstorming. I'd
prefer to add stuff incrementally, to make sure Perfherder can
handle the load, but I'd love to hear all your ideas.


Someone mentioned "test times" to me in private email.


That was me.  (I didn't feel like sending a late-at-night
one-sentence email to the whole list, and figured there was a decent
chance that somebody else would mention it as well.)

I think they're worth tracking because we've had substantial
performance regressions (I think including as bad as a doubling in
times) that weren't caught quickly, and led to substantially worse
load on our testing infrastructure.


I do think test times are worth tracking, but probably not in
Perfherder: test times might not be deterministic depending on
where / how they're running (which makes it difficult to
automatically detect regressions and sheriff them on a per commit
basis) and regardless there's too much data to really be manageable
by Perfherder's intended interface even if that problem were
magically solved.


It seems like if we're running the same tests on different sorts of
machines, we could track different perf numbers for the test run on
different machine classes.

We'd also want to measure the test time and *not* the time spent
downloading the build.

And we'd probably want to measure the total time across chunks so
that we don't count redistribution between chunks as a set of
regressions and improvements.

So that does make it a bit difficult, but it does seem doable.


As a possible alternative, I believe Kyle Lahnakoski's ActiveData
project (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData)
already *does* track this type of data but last I heard he was
looking for more feedback on how to alert/present it to the
platform community. If you have any ideas on this, please let him
know (he's CC'ed). :)


Perhaps, but I have no idea how to use it or what that would look
like.  The wiki page explicitly says it's for automated clients and
not by humans; it would be useful to see an example of such an
automated client to have an idea of how this would work.

-David



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: mozregression 2.0.0

[Andrew Halberstadt]

Just want to point out Julien also added a mach command wrapper for
mozregression that installs it and everything. Just add 'mach' in front
of his examples to try it out.

Thanks for all the work here Julien!

On 02/12/15 04:42 PM, Julien Pagès wrote:

Hello,

I'm happy to announce the release 2.0.0 of mozregression!

This is a major release, which uses a new bisection algorithm that makes it
much easier to use. Instead of forcing the user to specify a branch to
bisect against or defaulting to mozilla-inbound, mozregression now tries to
detect merge commits and will automatically switch to the correct branch to
continue the bisection after narrowing the range down to a day. As a
result, we can consolidate or eliminate many command line options.

Some examples of mozregression usage for bisecting Firefox:

# bisect using dates
mozregression -g 2015-11-20 -b 2015-11-25  # implied branch is m-c
mozregression -g 2015-11-20 -b 2015-11-25 --repo inbound
# bisect using changesets
mozregression -g dcd5230c4ce1 -b 931721112d8e  # implied branch is m-i
mozregression -g 1b2e15608f34 -b abbd213422a5 --repo m-c
# use debug builds
mozregression -g 2015-11-20 -b 2015-11-25 -B debug
mozregression -g dcd5230c4ce1 -b 931721112d8e -B debug
# launch a single build
mozregression --launch abbd213422a5 --repo m-c
mozregression --launch 2015-11-25 --repo aurora -B debug

See [1] for more details.

[1] http://mozilla.github.io/mozregression/news.html#2.0.0-release



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Andrew Halberstadt]

On 22/12/15 11:38 AM, Ben Kelly wrote:

On Tue, Dec 22, 2015 at 11:16 AM, Douglas Turner  wrote:


Mike -- totally supportive of this. I would *love* to see a release cycle
completely dedicated to quality.  We branch again on January 26.  We could
use that cycle to focus on nothing but quality (fixing tests, bug triaging,
no feature development at all).

Thoughts?



-1

This has been tried numerous times on fxos without success.  While it might
get oranges down temporarily, its not putting a system in place to address
the problem over the long haul.

I'd rather see us do:

1) Raise the visibility of oranges.  Post the most frequent intermittents
without an owner to dev-platform every N days.
2) Make its someone's job to find owners for top oranges.  I believe RyanVM
used to do that, but not sure if its still happening now that he has
changed roles.

Ben


FWIW a summary of top orangefactor[1] oranges are posted regularly
to dev.tree-alerts. Configuring it to also post to
dev.platform is certainly possible if that's what people want. Though I
have a feeling that people will mostly ignore these emails anyway if we do.

-Andrew

[1] http://brasstacks.mozilla.com/orangefactor/






On Tue, Dec 22, 2015 at 7:41 AM Mike Conley  wrote:


I would support scheduled time[1] to do maintenance[2] and help improve

our

developer tooling and documentation. I'm less sure how to integrate such

a

thing in practice.

[1]: A day, a week, heck maybe even a release cycle
[2]: Where maintenance is fixing oranges, closing out papercuts,
refactoring, etc.

On 21 December 2015 at 17:35,  wrote:


On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta

wrote:

So, I propose that we create an orangefactor threshold above which

the

tree should just be closed until people start fixing intermittent
oranges. Thoughts?

kats


How about regularly scheduled test fix days where everyone drops what

they

are doing and spends a day fixing tests? mc could be closed to

everything

except critical work and test fixes. Managers would be able to opt
individuals out of this as needed but generally everyone would be

expected

to take part.

Jim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Too many oranges!

[Andrew Halberstadt]

+1

I'd prefer to see quality be a perpetually ongoing effort over a
periodic burst. I'd like to see management rotate people into "quality
mode" by explicitly setting goals and deliverables around addressing it.

The problem in the past has been this notion of "greatest impact".
Things like refactorings and fixing tests always get deprioritized
because it's really hard to estimate how beneficial they are to the
overall mission. It was much easier to say shiny new feature X aligns
nicely with our top-line goals.

Well, finally, our top-line goal is to "build core strength". To me that
means that fixing tests and quality *now have the greatest impact*, and
everyone's deliverables should reflect this new reality.

-Andrew

On 22/12/15 11:40 AM, Bobby Holley wrote:

Do we actually need to have everybody working on oranges / quality at once?
Shouldn't we just prioritize it (probably permanently) in a way such that
_some_ members of every team are working on it?

Saying "everybody needs to work on this together" is a socially expedient
way of getting volunteers to do things, but we have a large paid staff and
management/planning structure that is designed to direct some of our
resources to walking and some of them to chewing gum.

Last spring I worked on media quality while other people worked on
features. It went great, and there was far less coordination overhead than
if everyone was trying to work on the same thing.

bholley

On Tue, Dec 22, 2015 at 8:32 AM, Mike Conley  wrote:


You had me at "quality". :)

On 22/12/2015 11:16 AM, Douglas Turner wrote:

Mike -- totally supportive of this. I would *love* to see a release
cycle completely dedicated to quality.  We branch again on January 26.
We could use that cycle to focus on nothing but quality (fixing tests,
bug triaging, no feature development at all).

Thoughts?

On Tue, Dec 22, 2015 at 7:41 AM Mike Conley mailto:mcon...@mozilla.com>> wrote:

 I would support scheduled time[1] to do maintenance[2] and help
 improve our
 developer tooling and documentation. I'm less sure how to integrate
 such a
 thing in practice.

 [1]: A day, a week, heck maybe even a release cycle
 [2]: Where maintenance is fixing oranges, closing out papercuts,
 refactoring, etc.

 On 21 December 2015 at 17:35, mailto:jmath...@mozilla.com>> wrote:

 > On Monday, December 21, 2015 at 1:16:13 PM UTC-6, Kartikaya Gupta
 wrote:
 > > So, I propose that we create an orangefactor threshold above
 which the
 > > tree should just be closed until people start fixing intermittent
 > > oranges. Thoughts?
 > >
 > > kats
 >
 > How about regularly scheduled test fix days where everyone drops
 what they
 > are doing and spends a day fixing tests? mc could be closed to
 everything
 > except critical work and test fixes. Managers would be able to opt
 > individuals out of this as needed but generally everyone would be
 expected
 > to take part.
 >
 > Jim
 > ___
 > dev-platform mailing list
 > dev-platform@lists.mozilla.org 
dev-platform@lists.mozilla.org>

 > https://lists.mozilla.org/listinfo/dev-platform
 >
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org 
dev-platform@lists.mozilla.org>

 https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: e10s tests

[Andrew Halberstadt]

I've been a little hesitant to plug this as it's a prototype and the UX
is pretty terrible. Also the underlying database it's using is not super
resilient. But people might find it useful for this effort and I don't
have time to improve it much more anyway, so what the heck.

This is basically what came of those "Test Informant Reports" I was
posting awhile back:
http://people.mozilla.org/~ahalberstadt/test-informant/

If you don't care about seeing how many tests were added or removed over
a period of time, set both the start and end dates to the same day.

Known issues:
1. Doesn't work over https (add an exception to https everywhere)
2. Setting either of the dates to a weekend or the current day may cause
weird results. This is because it's using data on the actual jobs that
got run that day. E.g if you set the end date to a Sunday and it so
happens that no PGO jobs were run, it'll interpret that as all PGO tests
getting disabled. It's a static client-side script, not an intelligent
web service.
3. Querying can be slooowww. Limit the number of suites.

-Andrew

p.s I'm terrible at webdev. Also I don't really have time for this. If
anyone is interested in improving/taking this over, let me know. Source
is here:
https://github.com/mozilla/test-informant


On 05/01/16 02:38 PM, Felipe G wrote:

(cross-posted to fx-dev and dev.platform)

As we drive towards shipping e10s, we are working on making sure that our
tests work with e10s. As you already know, there's a number of tests that
are disabled from running on e10s, and we need to enable them. We've
created a spreadsheet that lists every test that is disabled from running
on e10s in order to track the remaining work to do.

https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit?usp=sharing

Please take a look at the list and help in any way you can. Feel free to
add your name to any test that you pick to fix, or add comments about tests
that you have knowledge about. Even a comment describing what the test is
testing and/or what it takes to make it pass on e10s helps.  For tests that
can't be fixed in the timeframe of shipping e10s, we will consider manual
testing of the things that they are supposed to test. But of course that's
a stop gap solution until they can be properly fixed.

Blake and I will be reaching out to module owners and peers to find owners
for tests. It is each team's responsibility to ensure that their features
are properly tested under e10s! So do help and spread the word around.

There's a wiki page with tips on how to convert tests:
https://wiki.mozilla.org/Electrolysis/e10s_test_tips, and Blake has also
written about it recently:
https://blog.mozilla.org/mrbkap/2015/12/10/converting-a-test-to-be-e10s-compatible/
.

Ping me, mrbkap or anyone in #e10s if you have questions!

Felipe



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Changes to the firefoxtree hg extension

[Andrew Halberstadt]

Thanks for sharing this, I've seen a few people asking about it.

Note, as of bug 1234927 the `wip` alias that |mach mercurial-setup|
provides you has been fixed. But if you already had a pre-existing `wip`
alias, you'll have to update it manually like Patrick mentions.

-Andrew

On 11/01/16 06:00 AM, Patrick Brosset wrote:

I spent some time confronted to this last week, so for firefoxtree
extension users out there, this might be useful too.

The way the mercurial firefoxtree [1] extension maintains repository labels
(like "fx-team", "inbound", "central", "aurora", etc.) has changed recently
(not sure when this happened exactly, but I saw the change last week).
The labels used to be implemented as local mercurial tags (you'd see them
with `hg tags`).
Whenever you'd pull from, say, fx-team the "fx-team" tag would be moved to
the right revision. And you'd then be able to update to that revision
simply with `hg up fx-team`.

Updating to a given label still works but rather than using tags, the
extension now uses a new fxtree property. This property shows up in the
output of `hg log`.
If you recently ran `mach mercurial-setup`, you should have the latest
version of the firefoxtree extension, and therefore, the new fxtree
property (`hg log | grep fxtree` should output labels).

This change should be transparent to most users, except if you, like me,
use custom mercurial templates for some common operations like showing info
in a terminal prompt, or as a way to customize the output of `hg log` (I
use an alias called "wip" [2] which shows me parts of the graph with tags,
bookmarks, etc.).

You can add the new fxtree labels to your templates by using something
like:
{label("log.fxtree", if(fxtrees," {fxtrees}"))}

Hope this helps,
Patrick

[1]
http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html
[2] http://jordi.inversethought.com/blog/customising-mercurial-like-a-pro/



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: ESLint checks are now running on the nightly trees on checkin

[Andrew Halberstadt]

On 14/01/16 12:24 PM, Dave Townsend wrote:

All hail the mighty releng folks who have made it so ridiculously easy
to prototype and deploy new taskcluster tests!


I just want to emphasize the significance here. A developer unaffiliated
with either releng or the ateam and with only minimal support, just
stood up a new job in continuous integration, while only modifying files
that live in-tree.

A year or two ago, this would have been the quarterly goal of someone
who knew what they were doing. Big props to the taskcluster team!

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Just Autoland It

[Andrew Halberstadt]

On 22/01/16 12:20 AM, Nicholas Nethercote wrote:

On Thu, Jan 21, 2016 at 6:35 PM, Gregory Szorc  wrote:


I've gotten into the habit of just landing things if I r+ them and I think
they are ready to land. This has startled a few people because it is a major
role reversal of how we've done things for years. (Typically we require the
patch submitter to do the landing.) But I think reviewer-initiated landing
is a better approach


I often ask for review first and only do a try run after I've gotten
r+ and made any necessary changes. I also often make small changes
after obtaining review (e.g. minor comment fix-ups). Both of these
behaviours are incompatible with reviewer-initiated landing.

Maybe I'm unusual. But I can definitely understand why people are startled.


I do this too, I don't think it's unusual. Another case is when there
are multiple dependent bugs that need to land in a specific order, and
you flag them all for review at the same time for parallelism.

I was concerned at first as well, but after thinking about it some more,
this seems like something that will sort itself out automatically. There
may be a few miscommunications that lead to backouts at first, but in
the end either:

1. Patch authors will amend their workflows with the assumption that r?
== ready to land.

2. Specific teams/individuals will come to an agreement not to land each
others' patches. In this case if you need a review from someone outside
your team and you don't want them to land, a simple "Please do not land
yet" is likely good enough.

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Just Autoland It

[Andrew Halberstadt]

On 22/01/16 02:12 AM, Gregory Szorc wrote:

On Thu, Jan 21, 2016 at 10:37 PM, Mike Connor  wrote:


Like Greg, I'm a big fan of reviewer-lands-if-ready. It's a huge
simplification of workflow, saves developers time, and lets machines do
work instead of humans. That said, I don't think we should be surprising
people or unilaterally imposing changes to their workflow.  The best way to
do this is to make it simple to adopt, and demonstrably better than the old
way.  Developers will migrate over time as they see the light.

I think this is an easy fix if we're willing to modify our patch
submission workflow to reflect a wider set of asks and responses.  It's
more or less the same mental model as the multi-state flags we use for
tracking, there's more than one type of request, and more than one possible
response.

Simple proposal: replace review/feedback flags with a new,
multiple-requestable flag.  Possible values could be:

feedback?
review?
land?
feedback+
withcomments+
review+
land+





Patch authors would be able to opt into the easier flow by setting "land?"
instead of
"review?"  "review?" and "feedback?" would retain their current
definitions.  Patch reviewers would be able to respond explicitly with
feedback, a conditional r+, full r+, or land+.

* Anything where land+ is set would trigger autoland.
* land+ should be set only if requested, or if the reviewer believes
that's expected
* patch authors would be allowed to autoland with review+
* withcomments+ or feedback+ would not allow autoland.

In the short term, we could experiment with this as an additional patch
flag (land?/+/-).  I think the multi-state flag reflects current workflow
reality and eliminates nuance, so should be explored.



We've already talked about changing the Bugzilla flags to better express
intent. The BMO gurus say it isn't something we should undertake lightly.
It's much easier to experiment on the commit message / MozReview side of
things while leaving the existing Bugzilla flags in place. e.g. we can have
MozReview convert "land?" in commit messages to "review? flags in Bugzilla
while having autoland convert "land?" into "r=" upon landing.

It's worth noting that part of the reason we haven't enabled feedback flag
support and done more advanced things with commit messages is because
seemingly everyone practices slight variations of interactions with
Bugzilla flags and it's difficult codifying "One True Workflow" because
there are excellent justifications for nearly every variation. Code review
will continue to shift from Bugzilla centric to MozReview centric. And this
means that Bugzilla flags mean less and less over time. Perhaps we can
solve intent in MozReview without having to change anything in Bugzilla...


Personally I'm a little wary of adding too many flags. I think in this
case it should just be left as is. Will there be the occasional
miscommunication and backout as a result? Sure. That's not the end of
the world though.

People who work together often will quickly establish their personal set
of rules and expectations and adjust their workflows accordingly. And
for people who don't work together often, a simple "Please do not land
this yet" is only slightly less convenient than a flag, with the benefit
of being way more explicit.

In the end, it's on the reviewer. If the patch is complicated, has open
dependencies or looks like it might cause problems, as a reviewer I'm
not going to land it no matter what. If it's simple and self-contained,
why not?

-Andrew


On Fri, Jan 22, 2016 at 12:25 AM, Richard Newman 
wrote:


Both of these behaviours are incompatible with reviewer-initiated landing.




I've fallen on both sides of this particular fence; sometimes I want to
fire-and-forget a patch, and sometimes I still want to digest further after
getting review (or I know a piece of work is incomplete and further parts
will be forthcoming).

Perhaps this needs an opt-in flag on the input side?

"r?, and land it if you're happy" versus "r?, but I'll take care of
landing"?


___
firefox-dev mailing list
firefox-...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev






___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Just Autoland It

[Andrew Halberstadt]

On 22/01/16 10:30 AM, Boris Zbarsky wrote:

On 1/22/16 10:08 AM, Andrew Halberstadt wrote:

In the end, it's on the reviewer. If the patch is complicated, has open
dependencies or looks like it might cause problems, as a reviewer I'm
not going to land it no matter what. If it's simple and self-contained,
why not?


There is no obvious indication in mozreview whether a patch is
self-contained or has dependencies, is there?

Sometimes one can make a good guess, of course.

-Boris


Yeah not currently, you'd have to look at the bug. That might be a good
feature request. On the other hand, I'd feel uncomfortable landing
something for someone without looking at bug context anyway.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moving FirefoxOS into Tier 3 support

[Andrew Halberstadt]

On 25/01/16 11:00 AM, Fabrice Desré wrote:

   We're also working on a solution to move the b2g builds & tests to
their own infrastructure from which we'll ship our builds & updates.


What specifically does this mean? Do you mean infrastructure at the IT
level? Or at the continuous integration level (taskcluster,
mozharness, test harnesses, etc)?

If the latter, is there a detailed outline of the plans being proposed here?

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Just Autoland It

[Andrew Halberstadt]

On 25/01/16 05:44 PM, Eric Rescorla wrote:

On Mon, Jan 25, 2016 at 1:58 PM, Mike Hommey  wrote:


On Mon, Jan 25, 2016 at 11:23:59AM -0800, Steve Fink wrote:

Heh. Your list of UI complaints is very similar to mine. Some comments:


On 01/25/2016 04:26 AM, Honza Bambas wrote:

Writing both as a patch author and a reviewer as well.

- as a patch author I want a full control on when the patch actually

lands

(dependencies, any other timing reasons, that only the author knows the
best), "Don't land yet" comment somewhere will easily be overlooked
- as a reviewer I don't want to bare the decision to land or not to

land,

at all
- I want to preserve the mechanism "r+ with comments", which means to

let

the patch be landed by the author after updated (means reviewer doesn't
need to look over it again)
- as an author I want to write comments about the patch (about the
structure, what it does, why) to make the review simpler for the

reviewer

; commit message description may not be the best place, right?


Yes, is there a place for this? I feel this is a really important bit of
functionality that would fit naturally into the mozreview world, but I

don't

see how to do it. Situation: I put up a set of commits for review, and I
want to give per-commit notes to the reviewer that they'll see before
reviewing. Previously, I would put them in as comments on the bug and

either

pray that the reviewer happened to see them, or ping the reviewer on IRC

and

tell them to read them. In MozReview, I don't see a place to put such

things

at all?


It's also painful to use MozReview's comment system. The comments in the
reviews pane don't show much diff context, and while I just realized
it's possible to make it show more by hovering the diff part for a
little while, it's not really great, as it doesn't actually show a diff
view like the diff pane does, and switching to the diff pane a) is slow
for large diffs and b) has an entirely different comment UX that doesn't
seem really great either.



Indeed. It would be great if it would just include 5-8 lines of context by
default.

-Ekr


It's not terribly obvious, but instead of clicking on a line number you 
can click and drag on the numbers to set the exact amount of context you 
want.


-Andrew





Also, iirc, when you reply diff comments in MozReview, the resulting
comments sent to bugzilla lack context (but I could be misremembering).


- will it be possible to still be using hg patch queues?


A valid question, though I'd not that the mq-less workflow is actually
pretty good these days. mq is still easier/nicer for some things, but

doing

without mq is nicer in other ways, and the future leads away from mq.

On the other hand, there still isn't a great way to figure out the

mq-less

workflow. I remember a pretty good writeup from someone, and I intend to
write up my own. But there isn't anything that's easily discoverable.


Whether or not mq is deprecated, it still uses changesets that can be
pushed. And with a non publishing remote repository, it should still be
able to qpop without having to fool around with the draft state.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moving FirefoxOS into Tier 3 support

[Andrew Halberstadt]

Hi, I'm on the engineering productivity team, and work a lot on
continuous integration and test harnesses. I stood up initial Gecko
unittests on B2G emulator, worked on B2G automation for several years up
until the divide, and still help nominally maintain the emulator and
mulet unittests. So that's the hat I'll be wearing for this post.

I'd like to just state a few of the implications that this, and the
following quote from a gecko-all mail have:

This means that all engineering on Firefox OS by Platform
Engineering and Platform Operations will stop.


The purpose of this post is simply to make sure everyone involved is
aware of the following ramifications:

1. Gecko based test harnesses (mochitest, xpcshell, reftest, etc) will
eventually break and their corresponding jobs will be disabled. At some
point, the test harness will need to undergo a large enough change that
it will require a non-trivial amount of effort to not break B2G
emulators and mulet. I'd estimate for most harnesses, this will happen
sooner rather than later (within a quarter or two). When this happens,
the jobs will be turned off completely so as not to waste money on our
AWS bill. To be crystal clear, this means no more mochitest, reftest or
xpcshell on B2G emulators. Mulet will likely last a little longer as it
is similar enough to Firefox desktop.

2. If at any point CD wishes to rejoin mainline development and run the
set of Gecko unittests once again, re-integration will be a long and
difficult process.

3. Gaia tests will still need a substantial effort to keep green. This
one is more obvious, but still worth stating. It's really hard to keep a
job green after the fact. In my experience, keeping jobs in Tier 3 for
any extended period of time is not sustainable. CD will likely need to
fork if they want to keep these jobs green.

I think a question worth asking, is should we bother with Tier 3 at all?
Or should we jump straight to disabling CD specific jobs. I guess it
doesn't hurt to leave them running while they last, but in some cases
this will be a very short time frame.

Cheers,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Just Autoland It

[Andrew Halberstadt]

On 28/01/16 06:31 PM, Eric Rescorla wrote:

On Thu, Jan 28, 2016 at 10:58 AM, Gregory Szorc  wrote:


I'd like to thank everyone for the feedback in this thread. However, the
thread has grown quite long and has detoured from its original subject.

Speaking on behalf of everyone who works on MozReview, we know the
interface is lacking in areas and features are confusing or non-existent.
We're working on it. We're trying to morph workflows that have been
practiced at Mozilla for over a decade. We're playing a delicate balancing
game between giving familiarity with existing workflows (e.g. Bugzilla
integration) while trying to slowly nudge us towards more "modern" and more
powerful workflows.



Frankly, I'm a little dismayed to hear that you think that one of the goals
of Mozreview
is to modify people's workflows. The primary problem with our existing
review system
isn't that it doesn't involve some more "modern" review idiom (leaving
aside the question
of whether it is in fact more modern), but rather that the UI is bad and
that it's a lot
less powerful than existing review tools, including those that enact
basically the
same design (cf. Rietveld).

Speaking purely for myself, I'd be a lot happier if mozreview involved less
nudging
and morphing and more developing of basic functionality.

-Ekr


Not speaking to review per se, but engineering productivity in general.
The problem is there are so many unique and one-off workflows at Mozilla
that it gets harder and harder to improve "basic functionality" across
all of them. At some point, we hit vastly diminishing returns and get
stretched too thin. We'd love to improve every existing workflow, but
simply don't have the resources to do that.

Instead, we try to make one really nice workflow such that people want
to switch. That being said it's a valid opinion to think that the carrot
isn't sweet enough yet. If that's the case, filing bug reports like gps
mentioned is very helpful to us.

-Andrew



We're constantly surprised by all the one-off workflows and needs people
have and the reactions to a seemingly benign change. It's been a humbling
experience to say the least.

The best venue for reporting bugs, UX paper cuts, and suggest improvements
is Bugzilla. Developer Services :: MozReview. Or hop in #mozreview and chat
with us.

We get a lot of requests for changes that initially seem odd to us. So, if
your bug report could articulate why you want something and how many people
would benefit (e.g. "the layout team all does this"), it would help us
better empathize with your position and would increase the chances of your
request getting prioritized.


On Jan 28, 2016, at 10:14, Eric Rescorla  wrote:


On Thu, Jan 28, 2016 at 8:25 AM, Honza Bambas 

wrote:



On 1/28/2016 6:30, Karl Tomlinson wrote:

Honza Bambas writes:


On 1/25/2016 20:23, Steve Fink wrote:


For navigation, there's a list of changed files at the top
(below the fixed summary pane) that jumps to per-file anchors.

Not good enough for review process.

Are you saying you want tabs or something for this (like

splinter uses)? I'd certainly like something less sluggish, but
maybe that's just my browser again.

Yes please.  Having one file on the screen at a time is very
useful.

The next/previous file/comment keyboard shortcuts may be useful in
the meantime.


Unfortunately not.  The intention is that when I scroll down the screen
I'm at the end of *a single file*, and of course up the screen means to

be

up at that same file.  Shortcuts are definitely unhelpful for me.  With

how

revboard works now it's just a mess of all  put together.


I agree with this. As I've mentioned before, NSS uses Rietveld, which
file-by-file, and I find
this much more convenient.

-Ekr



Thanks anyway!

-hb-







https://www.reviewboard.org/docs/manual/2.5/users/reviews/reviewing-diffs/#keyboard-shortcuts

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reftests moving to structured logging

[Andrew Halberstadt]

Reftest is the last major test harness still not using structured logs,
but that should change by the end of the week. See bug 1034290 [1] for
more details.

I've tried my best to make sure things like reftest-analyzer,
leak/assertion checks, crash detection, etc. all continue to work. But
due to the sad lack of tests for the harnesses themselves, it's possible
that I missed something. So if you see anything not working like it
should, please file a bug blocking bug 1034290 [1] and CC me.

What does this change mean for reftest? In the short term, nothing
should be different save that reftests will start working with tools
that depend on structured logging (e.g ActiveData, auto-starring, etc).
In the medium term, it means we'll be able to tweak the log format
without breaking anything (once consumers that are still parsing the
formatted log get updated). In the long term, structured logging will be
a foundation upon which new data driven tools will be built.

Let me know if you have any questions or concerns,
-Andrew

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1034290
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reftests moving to structured logging

[Andrew Halberstadt]

This is now live on central.

On 04/02/16 01:28 PM, Andrew Halberstadt wrote:

Reftest is the last major test harness still not using structured logs,
but that should change by the end of the week. See bug 1034290 [1] for
more details.

I've tried my best to make sure things like reftest-analyzer,
leak/assertion checks, crash detection, etc. all continue to work. But
due to the sad lack of tests for the harnesses themselves, it's possible
that I missed something. So if you see anything not working like it
should, please file a bug blocking bug 1034290 [1] and CC me.

What does this change mean for reftest? In the short term, nothing
should be different save that reftests will start working with tools
that depend on structured logging (e.g ActiveData, auto-starring, etc).
In the medium term, it means we'll be able to tweak the log format
without breaking anything (once consumers that are still parsing the
formatted log get updated). In the long term, structured logging will be
a foundation upon which new data driven tools will be built.

Let me know if you have any questions or concerns,
-Andrew

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1034290


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Generic Task Cluster Tasks / File Based Task Scheduling

[Andrew Halberstadt]

On 17/02/16 01:59 PM, Gregory Szorc wrote:

* You can now only run tasks if certain files changed. This opens the door
for drastic reduction in automation load via more intelligent scheduling of
tasks via in-tree configurations. (For the curious, relevant commits/files
are determined from a combination of the Mercurial pushlog and phases, so
pushes to Try not pushing all changesets will still pull in changes from
draft changesets previously pushed to Try.) See
https://hg.mozilla.org/integration/mozilla-inbound/rev/eee2e3b43fc1 for
more on this feature.

As part of this, the eslint task has been converted to a generic task and
now only runs if certain files change. This means e.g. C++ only pushes
shouldn't trigger eslint tasks.


Somewhat related, we should also only lint the files that were touched
by a given push. This ties in to the mozlint proposal I made awhile
back. I should really try and polish that up and get something landed.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Test automation addons and signing

[Andrew Halberstadt]

To date, our continuous integration has been setting
'xpinstall.signatures.required=false' to bypass addon signing. But soon,
this pref will become obsolete and Firefox will enforce signing no
matter what.

In preparation, we will begin signing extensions that are used in our
test automation. If you maintain one of these addons, it means you will
be required to re-sign it everytime a change is made. For more
information on what this means, how to sign addons or how to bypass
signing completely, see the following document:
https://wiki.mozilla.org/EngineeringProductivity/HowTo/SignExtensions

Let me know if you have any questions or concerns,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Exit code -11 must die

[Andrew Halberstadt]

On 29/02/16 12:03 PM, Benjamin Smedberg wrote:

On 2/27/2016 9:06 PM, Randell Jesup wrote:

months until recently it popped up a bit).  Note that this failure
*never* results in a crashdump, and I've never seen it locally, just in
Automation.


What we do know:

  * Exit code -11 is evidence a SIGSEGV (crash).

This I don't know, but somebody may know (+ted):

  * Are we sure that the crash is happening in firefox.exe? Or is it
possible that some other process is crashing and taking down our
test harness with it?
  * Can somebody point to exactly what line of code in the test harness
collects the -11 code?


Returning -signal is an idiom from python's subprocess stdlib that
mozprocess replicates:
https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#593



  * Is there no crash dump because the crash reporter is turned off?
  o If it's turned on, is the crash reporter itself crashing? Or is
the test harness not picking up the crash dump?


I've seen shutdown crashes that don't generate dumps before. Like bug
1228595. Ted theorized that the chrome process is too far into shutdown
to handle the crash properly.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: firefox-ui-tests are now in mozilla-central with TaskCluster support

[Andrew Halberstadt]

They're more like end-to-end tests than browser-chrome. E.g instead of
calling Gecko APIs, they'll send a mouse event to a button, send key
events to a textbox, etc. So they'll theoretically perform actions
closer to what a user might do.

Since they're written in python, they can also do things like restart
the browser within the test (for e.g testing updates). It uses a library
called Firefox Puppeteer that's built on top of marionette-client:
http://firefox-puppeteer.readthedocs.org/en/latest/

(which lives in testing/puppeteer despite what the docs say)

Henrik can probably go into more details on what is tested.

-Andrew

On 01/03/16 12:18 PM, Matthew N. wrote:

CC: firefox-dev

On 2016-03-01 5:17 AM, Henrik Skupin wrote:

Over the last years the formerly known Mozmill tests and now Firefox ui
tests have been located in their own repository. That meant that we
always got regressions due to changes developers have been made in
Firefox. Hunting down those regressions and fixing them was always a
time consuming job. Beside all that we were also responsible for merging
our branches accordingly to our 6 week cycle.


Hi Henrik, thanks for sharing.

Could you give an overview of what is tested by this suite and how it
differs from mochitest-browser-chrome? It seems one difference is that
some(?) tests run on non-en-US.

Could you also add a description to
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing?

Thanks,
Matthew N. (:MattN)


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Try syntax is no longer required when pushing to try

[Andrew Halberstadt]

With treeherder's "Add New Jobs" [1] UI, using try syntax is no longer
the only way to schedule stuff on try. As of now, it's possible to push
to try without any try syntax. If you do this, no jobs will be scheduled
on your push and it will be up to you to add whichever jobs you want via
"Add New Jobs". Please note, it's not currently possible to schedule
taskcluster jobs this way. Taskcluster support will be tracked by bug
1246167 [2].

Otherwise using try syntax should work just the same as before. For more
information and detailed instructions, see:
https://wiki.mozilla.org/Try

Special thanks to adusca and armenzg for pushing on "Add New Jobs".

-Andrew

[1] https://wiki.mozilla.org/Try#Scheduling_jobs_with_Treeherder
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1246167
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Try syntax is no longer required when pushing to try

[Andrew Halberstadt]

On 03/03/16 12:37 PM, Andreas Tolfsen wrote:

On 3 March 2016 at 17:25, Andrew Halberstadt  wrote:

With treeherder's "Add New Jobs" [1] UI, using try syntax is no longer
the only way to schedule stuff on try. As of now, it's possible to push
to try without any try syntax. If you do this, no jobs will be scheduled
on your push and it will be up to you to add whichever jobs you want via
"Add New Jobs".


This is great!

It would be even greater to see this integrated with MozReview some
day, so that when you request a try run through a review’s Automation
menu, it would present you with a similar diagram for what jobs to
include.

Is the Add New Jobs functionality able to figure out the dependencies
of a chosen target?  If I request a test to run on Mac OS X 10.7 but I
haven’t built on that platform, will it also trigger the build job?


Yes, it will trigger the build automatically if you select a test job
and the build doesn't already exist.

A cheap win for mozreview, might be to redirect to the "Add New Jobs"
view in treeherder if the try syntax was left blank.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Intent to enable e10s by default when running tests locally

[Andrew Halberstadt]

Bug 1243083 tracks enabling e10s by default when running tests locally.
I intend to do this for as many harnesses as possible unless someone
says otherwise.

The implication is that the --e10s flag will be renamed to
--disable-e10s. So you'll need to pass in --disable-e10s if you wish to
run without e10s enabled. This also means that tests that are skipped
with 'e10s' will no longer run by default locally. The hope is that this
will make it easier for devs to find and fix tests that are broken with
e10s enabled, as well as ensure that new tests work with e10s right out
of the gate. CI jobs will not be affected, only running locally.

One potential sticking point is mochitest-chrome. It currently does not
get run in e10s in CI, so local configuration should mirror this.
However, since --e10s is being removed, this means it would be
impossible to run mochitest-chrome in e10s without modifying source
files. Maybe this just needs some argparse hackery.

If you have any concerns or know of other suites that should explicitly
*not* be run with e10s enabled by default, please let me know.

Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to enable e10s by default when running tests locally

[Andrew Halberstadt]

I'm not aware of work around this. If --debugger is completely busted
with e10s, I could potentially make --debugger imply --disable-e10s
until it gets fixed. Is there a bug on file?

I also forgot to mention that command defaults are likely coming soon,
so once bug 1255450 lands you'll be able to make a .machrc with:

[defaults]
mochitest = --disable-e10s


On 24/03/16 12:30 PM, Boris Zbarsky wrote:

On 3/24/16 12:15 PM, Andrew Halberstadt wrote:

If you have any concerns or know of other suites that should explicitly
*not* be run with e10s enabled by default, please let me know.


Do we have any plans for making --debugger not completely useless when
running tests in e10s mode?  There are various ways of hacking around
it, but it would be awesome if we just made it work by default somehow...

-Boris


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to enable e10s by default when running tests locally

[Andrew Halberstadt]

Afaict, ./mach mochitest -f chrome --e10s enables e10s. At least the
python side isn't overriding that default. Maybe there's some JS code
somewhere that overrides it.

But if the intent is for mochitest-chrome to never run with e10s enabled
anyway, then I guess not having that option isn't a big deal.

On 24/03/16 01:25 PM, Andrew McCreight wrote:

On Thu, Mar 24, 2016 at 9:15 AM, Andrew Halberstadt <
ahalberst...@mozilla.com> wrote:


Bug 1243083 tracks enabling e10s by default when running tests locally.
I intend to do this for as many harnesses as possible unless someone
says otherwise.



Great!

[some text deleted]

One potential sticking point is mochitest-chrome. It currently does not

get run in e10s in CI, so local configuration should mirror this.
However, since --e10s is being removed, this means it would be
impossible to run mochitest-chrome in e10s without modifying source
files. Maybe this just needs some argparse hackery.



My impression was that mochitest-chrome doesn't actually run with e10s,
even when you specify the flag. Is that not correct?



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to enable e10s by default when running tests locally

[Andrew Halberstadt]

My understanding is shipping e10s is the top priority, so I believe that
implies running tests there is (slightly) favoured.

But I like your idea for a dual mode. I'm on the fence whether it would
be a good default or not as it will double the time to run tests
locally, and many people likely don't need to test both (as try will do
it for them). This might be a good use case for command aliases (landing
soon in bug 1255450 hopefully).

Andrew

On 03/04/16 11:51 PM, Blair McBride wrote:

Default options imply the default is somehow favoured over the
non-default. Which, for this, makes me wonder: Is getting tests passing
on non-e10s less important now?

I've found the ergonomics around testing both to be quite a footgun -
it's a manual process to run tests under both modes, and it's too easy
to forget to run tests under the other mode. Would be handy to have a
switch to enable running under both modes (or even better, defaulting to
both modes).

- Blair



Andrew Halberstadt wrote:

Bug 1243083 tracks enabling e10s by default when running tests locally.
I intend to do this for as many harnesses as possible unless someone
says otherwise.

The implication is that the --e10s flag will be renamed to
--disable-e10s. So you'll need to pass in --disable-e10s if you wish to
run without e10s enabled. This also means that tests that are skipped
with 'e10s' will no longer run by default locally. The hope is that this
will make it easier for devs to find and fix tests that are broken with
e10s enabled, as well as ensure that new tests work with e10s right out
of the gate. CI jobs will not be affected, only running locally.

One potential sticking point is mochitest-chrome. It currently does not
get run in e10s in CI, so local configuration should mirror this.
However, since --e10s is being removed, this means it would be
impossible to run mochitest-chrome in e10s without modifying source
files. Maybe this just needs some argparse hackery.

If you have any concerns or know of other suites that should explicitly
*not* be run with e10s enabled by default, please let me know.

Andrew
___
firefox-dev mailing list
firefox-...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to enable e10s by default when running tests locally

[Andrew Halberstadt]

On 04/04/16 09:37 AM, Ehsan Akhgari wrote:

On Sun, Apr 3, 2016 at 11:51 PM, Blair McBride  wrote:


Default options imply the default is somehow favoured over the
non-default. Which, for this, makes me wonder: Is getting tests passing on
non-e10s less important now?



Fennec doesn't use e10s, so at least for tests that cover both desktop and
Android, they're both equally important.  Plus, as I understand the plans,
we're going to be shipping e10s and non-e10s on desktop at the same time to
different users at least for a while.


The default on Fennec will remain non-e10s. And yes, running non-e10s is
still important on desktop too. But if you *had* to pick one or the other..

I filed a bug [1] to at least implement a dual mode. We can debate
whether it should be default or not later.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1261823



I've found the ergonomics around testing both to be quite a footgun - it's
a manual process to run tests under both modes, and it's too easy to forget
to run tests under the other mode. Would be handy to have a switch to
enable running under both modes (or even better, defaulting to both modes).

- Blair




Andrew Halberstadt wrote:


Bug 1243083 tracks enabling e10s by default when running tests locally.
I intend to do this for as many harnesses as possible unless someone
says otherwise.

The implication is that the --e10s flag will be renamed to
--disable-e10s. So you'll need to pass in --disable-e10s if you wish to
run without e10s enabled. This also means that tests that are skipped
with 'e10s' will no longer run by default locally. The hope is that this
will make it easier for devs to find and fix tests that are broken with
e10s enabled, as well as ensure that new tests work with e10s right out
of the gate. CI jobs will not be affected, only running locally.

One potential sticking point is mochitest-chrome. It currently does not
get run in e10s in CI, so local configuration should mirror this.
However, since --e10s is being removed, this means it would be
impossible to run mochitest-chrome in e10s without modifying source
files. Maybe this just needs some argparse hackery.

If you have any concerns or know of other suites that should explicitly
*not* be run with e10s enabled by default, please let me know.

Andrew
___
firefox-dev mailing list
firefox-...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev



___
firefox-dev mailing list
firefox-...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev







___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Intent to enable e10s by default when running tests locally

[Andrew Halberstadt]

This has now landed on central. The following suites are e10s by default
and require the --disable-e10s flag to run non-e10s:

* marionette
* mochitest (excluding chrome, a11y and android)
* reftest (excluding android)
* talos
* web-platform-tests

I also added some logging to marionette, mochitest and reftest both at
the start and end of the test run to make it easier to see whether e10s
was enabled or not (ctrl-f for e10s).

Please file bugs blocking bug 1243083 if you believe something is amiss.
Finally, bug 1261823 was filed to investigate a dual-mode that runs both
e10s and non-e10s. There is no eta at this time.

-Andrew

On 24/03/16 12:15 PM, Andrew Halberstadt wrote:

Bug 1243083 tracks enabling e10s by default when running tests locally.
I intend to do this for as many harnesses as possible unless someone
says otherwise.

The implication is that the --e10s flag will be renamed to
--disable-e10s. So you'll need to pass in --disable-e10s if you wish to
run without e10s enabled. This also means that tests that are skipped
with 'e10s' will no longer run by default locally. The hope is that this
will make it easier for devs to find and fix tests that are broken with
e10s enabled, as well as ensure that new tests work with e10s right out
of the gate. CI jobs will not be affected, only running locally.

One potential sticking point is mochitest-chrome. It currently does not
get run in e10s in CI, so local configuration should mirror this.
However, since --e10s is being removed, this means it would be
impossible to run mochitest-chrome in e10s without modifying source
files. Maybe this just needs some argparse hackery.

If you have any concerns or know of other suites that should explicitly
*not* be run with e10s enabled by default, please let me know.

Andrew


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Mach command aliases and runtime configuration

[Andrew Halberstadt]

Hey all, bug 1255450 has landed which means..

For mach users:
You can now create a machrc (or .machrc) file in the following locations:
* $MACHRC
* ~/.mozbuild
* topsrcdir

In the future, individual commands may implement their own settings, but
for now a single section called 'alias' is implemented in mach core. You
can use 'alias' to provide defaults, or create an entirely new command, e.g:

[alias]
mochitest = mochitest --disable-e10s
browser = mochitest -f browser

To see a list of all implemented settings, you can run:
../mach settings -l


For mach command developers:
You can now provide configuration for your command. See here for
information how:
http://gecko.readthedocs.org/en/latest/python/mach/settings.html#defining-settings

Let me know if anything isn't working as expected.

-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: One Firefox repository to rule them all

[Andrew Halberstadt]

This is really cool!

Though I much prefer firefoxtree's namespace updating to keep track of
remote heads over using bookmarks. I want a label that will always point
to the last known head on the server, so e.g
`hg update central && hg commit -m "Foo"` should not move 'central'.
Using bookmarks to track the remote heads is also incompatible with my
bookbinder extension which I've come to rely quite heavily on. This
would be a personal blocker for me to make the switch.

Maybe firefoxtree could be adapted to work with this new repo as well.
Or maybe I could look into doing something with remotenames.

Andrew



On 14/04/16 08:22 PM, Gregory Szorc wrote:

I'm pleased to announce the immediate availability of some *experimental*
read-only Mercurial repositories containing the combined, useful history of
the various Firefox repositories, all in chronological order and stored in
a more efficient format that is faster to clone and pull from and results
in faster client operations.

The repositories can be found at https://hg.mozilla.org/experimental. The
repository you likely want to clone is
https://hg.mozilla.org/experimental/firefox-unified. A visualization
showing the chronological history of the repo can be seen at
https://hg.mozilla.org/experimental/firefox-unified/graph.

The primary goal of these repositories is to provide developers (and
eventually automation) with more efficient interaction with the Firefox
source repositories. There are several secondary and side-benefits,
including improving the scalability of Try and MozReview's repositories.

More documentation about these repos is available at [1]. tl;dr

* The repositories contain all the commits from the Firefox repositories
you use everyday (central, inbound, fx-team, aurora, beta, esr, etc).
* The repositories do not contain all the *_RELBRANCH branches (which
basically have no value to the average developer).
* Thes unified repositories are ~300MB *smaller* than mozilla-central
despite containing ~28,000 more commits. This was achieved through light
magic.
* Mercurial bookmarks are used to track the heads of the various Firefox
repos.
* The pushlog data is derived from the first known push of a changeset, so
it should match what's on e.g. central, inbound, etc.
* Sadly, git-cinnabar won't be able to talk to these repos just yet due to
git-cinnabar not supporting some modern Mercurial features. A GitHub issue
is on file at [2].

If you use the "firefoxtree" extension to manage a unified repository
today, you should consider switching to one of these new unified
repositories instead: it should be faster and easier to reason about.

The repositories have the "experimental" label attached so we can reserve
the right to make changes without people complaining too loudly about
backwards compatibility. (But I wouldn't worry too much about stability -
I'm committed to keeping these running and improving them.) The goal is to
flush out issues with these repositories then remove the "experimental"
label. After that, we can have automation start consuming these
repositories. After that, we can perhaps start thinking about consolidating
around a single, canonical repository, including pushing. But that's a
topic for another day.

I'm very anxious for feedback on these repositories. Please make noise in
dev-version-cont...@lists.mozilla.org, #vcs, the "Developer Services:
Mercurial: hg.mozilla.org" bug component, or in bug 1108729.

[1]
https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html
[2] https://github.com/glandium/git-cinnabar/issues/64



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Searchfox (new code search tool)

[Andrew Halberstadt]

On 07/06/16 01:18 AM, Mike Hommey wrote:

I'm going to sound negative, but why? Or more precisely, why not
contribute to DXR to add those features that you implemented in
searchfox that DXR doesn't have?

MXR is already taking too long to fade out of existence, do we really
want yet another different tool?

Mike


+1. The technical achievement here is very impressive, and there looks
to be a ton of useful features. The blame walking looks especially
useful for me. But DXR is very actively developed, has 45 contributors
and 146 forks:
https://github.com/mozilla/dxr

I just don't see the point of competition between internal code
searching tools. I would love if ideas from searchfox (like blame
walking) could be incorporated into DXR.

Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

ESlint infrastructure moved to 'tools/lint'

[Andrew Halberstadt]

If you had any absolute paths in editor configuration, you'll need to
update them to point to 'tools/lint/eslint'.

This change was in preparation for getting |mach eslint| integrated with
the mozlint library, see bug 1258341 for details. My aim will be to
maintain backwards compatibility both in function and UX, but if you
have questions or concerns about this upcoming change, please reply
here, in bug or via e-mail.

Thanks,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rust required to build Gecko

[Andrew Halberstadt]

When first installing rust with ./mach bootstrap the install is
successful, but there is a message about not being able to find the
compiler immediately afterwards.

For anyone confused by this, the binaries are downloaded to ~/.cargo/bin
and adding this directory to your $PATH should fix the issue. The
bootstrapper explains this if you run it a second time, but makes no
mention of it the first time through for some reason.

-Andrew

On 24/11/16 05:49 PM, Ralph Giles wrote:

tl;dr This is a heads-up that all gecko developers should install rust.

Next week I plan to switch our default build config to require Rust
when building Firefox.[1] If you compile Firefox from the C++ source,
please install the Rust language environment now.

You can install Rust by running `./mach bootstrap` which will download
and run the rustup installer for you.[2]

We recommend the installer at https://rustup.rs/ (despite being beta)
because it makes staying up to date and cross-compilation easy. If you
want more control, or to experiment with rust, you can install
directly from that website.

The main thing is to have up-to-date versions of the `rustc` and
`cargo` executables in the path of your build shell. Rust releases
every six weeks, just like Firefox, and we generally require the
latest stable release to compile mozilla-central. You can stay current
by running `rustup update`.

You'll still be able to build without a rust compiler by adding:

  ac_add_options --disable-rust

to your mozconfig. This is a temporary work-around; we expect to
remove that option and require Rust unconditionally early next year as
non-optional features start to depend on it.

Rust language in Gecko is an important part of Project Quantum. I'm
excited to be taking this next step toward that future. We first
shipped Rust code to users in Firefox 48, so it's long past time this
aspect of our default builds matched what we release.[3]

Thanks for your attention,
 -r

[1] Enabling rust is https://bugzil.la/1283898
[2] bootstrap support landed Tuesday in https://bugzil.la/1286799
[3] If you have issues with the installer or build, please file issues
blocking our tracking bug at https://bugzil.la/oxidation



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Several leak failures have slipped passed continuous integration

[Andrew Halberstadt]

Over the holidays, we noticed that leaks in mochitest and reftest were 
not turning jobs orange, and that the test harnesses had been running in 
that state for quite some time. During this time several leak related 
test failures have landed, which can be tracked with this dependency tree:

https://bugzilla.mozilla.org/showdependencytree.cgi?id=1325148&hide_resolved=0

The issue causing jobs to remain green has been fixed, however the known 
leak regressions had to be whitelisted to allow this fix to land. So 
while future leak regressions will properly fail, the existing ones (in 
the dependency tree) still need to be fixed. For mochitest, the 
whitelist can be found here:

https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#2218

Other than that, leak checking is only disabled on linux crashtests.

Please take a quick look to see if there is a leak in a component for 
which you could help out. I will continue to help with triage and 
bisection for the remaining issues until they are all fixed. Also big 
thanks to all the people who are currently working on a fix or have 
already landed a fix.


Read on only if you are interested in the details.


_Why wasn't this caught earlier?
_
The short answer to this question is that we do not have adequate 
testing of our CI.

_
_The problem happened at the intersection between mozharness and the 
test harnesses. Basically a change in mozharness exposed a latent bug in 
the test harnesses, and was able to land because it appeared as if 
nothing went wrong. Catching errors like this is tricky because regular 
unit tests would not have detected it either. It requires integration 
tests of the CI system as a whole (spanning test harnesses, mozharness 
and buildbot/taskcluster).



_How will we prevent this in the future?_

Historically, integration testing our test harnesses has been a hard 
problem. However with recent work in taskcluster, python tests and some 
refactoring on the build frontend, I believe there is a path forward 
that will allow us to stand up this kind of test. I will commit some of 
my time to fix this and hope to have /something/ running that would have 
caught this by the end of Q1.


I would also like to stand up a test harness designed to test command 
line applications in CI, which would provide another avenue for writing 
test harness unit and integration tests. Bug 1311991 
 will track this work.


It is important that developers are able to trust our tests, and when 
bugs like this happen, that trust is eroded. For that I'd like to 
apologize, and express my hope that this will be the last time a major 
test result bug like this happens again. At the very least, we need to 
have the capability of adding a regression test when a bug like this 
happens in the future.


Thanks for your help and understanding.
- Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Several leak failures have slipped passed continuous integration

[Andrew Halberstadt]

Ah, I misunderstood.

If the log line is printed or dumped directly to stdout, then the string
TEST-UNEXPECTED-FAIL actually will fail the job. It's only if we do
log.info('TEST-UNEXPECTED-FAIL ...') that we run into problems. So if
you see the string 'TEST-UNEXPECTED-FAIL' somewhere, it isn't
necessarily wrong (though I think we should stop relying on this string
anyway).

I did a quick DXR for the failure case and found one place in a rarely
used mozrunner utility function that needs to be updated. Afaict, no
problems have landed as a result, but will get this fixed asap as well.


On 29/12/16 02:55 PM, Boris Zbarsky wrote:

On 12/29/16 11:49 AM, ahalberst...@mozilla.com wrote:

Have we done any sort of audit to see whether any other tests got broken
by the structured logging changes?  Had we done such an audit after bug
1321127 was fixed?


Once bug 1321127 was fixed, any other tests that were broken would
turn the job orange so would have prevented the fix from landing.


No, what I meant was this: we now have two separate instances of
harnesses not showing failures because they were relying on the
TEST-UNEXPECTED-FAIL string parsing, not the structured logger.  Have we
done any auditing for _other_ cases that are relying on said string
parsing?

-Boris

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Test harness preferences moving out of automation.py

[Andrew Halberstadt]

tl;dr - If you update a test harness pref in build/automation.py.in, 
please also update it in testing/profiles/prefs_general.js. This is 
temporary and I am working to remove the automation.py.in location in 
bug 830430[1]


---

Currently prefs that are common to our test harnesses are hardcoded in a 
string of Javascript inside of automation.py. This is bad for many reasons:


1) It isn't obvious to find - as a result prefs for various harnesses 
have been littered throughout the tree
2) Most of the newer harnesses don't use automation.py - these have no 
way of taking advantage of these prefs and must duplicate them
3) Storing Javascript in a python string and writing to a file is just 
yucky in general - it should start out in a js file


Instead we have decided to create a single canonical directory that will 
store all prefs (and other profile information). This location is 
testing/profiles.


So, if you update a test harness pref in build/automation.py.in, please 
also update it in testing/profiles/prefs_general.js. Having two 
locations for prefs is bad, and I'm working to remove the automation.py 
location in bug 830430[1]. I'll make sure the two locations are in sync 
before doing so.


If you have any questions, concerns or suggestions feel free to contact 
me or post in the bug.


Thanks,
Andrew

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=830430

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Test harness preferences moved out of build/automation.py.in to testing/profiles/prefs_general.js

[Andrew Halberstadt]

As part of a broader automation refactor, we are consolidating all of 
the test harness preferences in one place. Storing them as a string in a 
python file is less than ideal, so as a first step the preferences in 
automation.py.in have been moved to testing/profiles/prefs_general.js [1].


Note that only mochitest, leaktest and profileserver used these prefs. 
Reftest and xpcshell have their own prefs in separate locations which 
will also be moving in the future.


Consolidating preferences into files will allow us to be more flexible 
with which preferences get used by which harness / build configurations. 
It will also make it easier to find for people unfamiliar with the 
various harnesses.


Thanks,
Andrew

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=830430
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Andrew Halberstadt]

I used to also really dislike mercurial, but the more I used it the more 
I started to realize its power. Gps summarized it much better than I 
ever could have in a blog post last month:


http://gregoryszorc.com/blog/2013/05/12/thoughts-on-mercurial-%28and-git%29/

I think mercurial and git are both great, but maybe the easier way to 
solve this problem is through better mercurial education.


Andrew

On 05/30/2013 08:56 PM, Johnny Stenback wrote:

[TL;DR, I think we need to embrace git in addition to hg for
Firefox/Gecko hacking, what do you think?]

Hello everyone,

The question of whether Mozilla engineering should embrace git usage for
Firefox/Gecko development has come up a number of times already in
various contexts, and I think it's time to have a serious discussion
about this.

To me, this question has already been answered. Git is already a reality
at Mozilla:

1. Git is in use exclusively for some of our significant projects (B2G,
Gaia, Rust, Servo, etc)
2. Lots of Gecko hackers use git for their work on mozilla-central,
through various conversions from hg to git.

What we're really talking about is whether we should embrace git for
Firefox/Gecko development in mozilla-central.

IMO, the benefits for embracing git are:

   * Simplified on-boarding, most of our newcomers come to us
 knowing git (thanks to Github etc), few know hg.
   * We already mirror hg to git (in more ways than one), and
 git is already a necessary part of most of our lives.
 Having one true git repository would simplify developers'
 lives.
   * Developers can use git branches. They just work,
 and they're a good alternative to patch queues.
   * Developers can benefit from the better merge algorithms
 used by git.
   * Easier collaboration through shared branches.
   * We could have full history in git, including all of hg
 and CVS history since 1998!
   * Git works well with Github, even though we're not switching
 to Github as the ultimate source of truth (more on that below).

Some of the known issues with embracing git are:

   * Performance of git on windows is sub-optimal (we're
 already working on it).
   * Infrastructure changes needed...

So in other words, I think there's significant value in embracing git
and I think we should make it easier to hack on Gecko/Firefox with git.
I see two ways to do that:

1: Embrace both git and hg as a first class DVCS.
2: Switch wholesale to git.

Option 1 is where I personally think it's worth investing effort. It
means we'd need to set up an atomic bidirectional bridge between hg and
git (which I'm told is doable, and there are even commercial solutions
for this out there that may solve this for us). Assuming we solve the
bridge problem one way or another, it would give us all the benefits
listed above, plus developer tool choice, and we could roll this out
incrementally w/o the need to change all of our infrastructure at once.
I.e. our roll out could look something like this:

1. create a read only, official mozilla-central git mirror
2. add support for pushing to try with git and see the results in tbpl
3. update tbpl to show git revisions in addition to hg revisions
4. move to project branches, then inbound, then m-c, release branches, etc

While doing all this, things like build infrastructure and l10n would be
largely, if not completely, unaffected. Lots of details TBD there, but
the point is we'd have a lot of flexibility in how we approach this
while the amount of effort required before our git mirror is functional
will be minimal compared to doing a wholesale switch as described below.
We would of course need to run high availability servers for both hg and
git, and eventually the atomic bidirectional bridge (all of which would
likely be on the same hardware).

Option 2 is where this discussion started (in the Tuesday meeting a few
weeks ago,
https://wiki.mozilla.org/Platform/2013-05-07#Should_we_switch_from_hg_to_git.3F).
Since then I've had a number of conversations and have been convinced
that a wholesale change is the less attractive option. The cost of a
wholesale change will be *huge* on the infrastructure end, to a point
where we need to question whether the benefits are worth the cost. I
have also spoken with other large engineering orgs about git performance
limitations, one of which is doing the opposite switch, going from git
to hg. While I don't see us hitting those limitations any time soon, I
also don't think the risk of hitting those limitations is one we want to
take in a wholesale change at this point.

One inevitable question that will arise here if we were to switch
wholesale over to git is whether we're also considering hosting
Firefox/Gecko development on Github, and the answer to that question at
this point is no (but we will likely continue to mirror mozilla-central
etc to Github). We've been in talks with Github, but we will not get the
reliability guarantees we need nor the flexibility we need if we were to
hos

Mach for b2g emulator unittests

[Andrew Halberstadt]

Running mochitests/reftests/xpcshell on a B2G emulator has been a bit of 
a struggle in the past. As of now, if you have an emulator configured 
and built, you can simply run './mach mochitest-remote' (or ./mach help 
mochitest-remote for additional arguments). Substitute 'mochitest' with 
'reftest', 'crashtest' or 'xpcshell' to run those suites.


For more details, see my blog post: 
http://ahal.ca/blog/2013/running-b2g-unittests-mach/


Don't hesitate to ping me if you have problems or questions.

Cheers,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Poll: What do you need in MXR/DXR?

[Andrew Halberstadt]

Better python support. For example, the function name parameter doesn't 
work with ext: .py


http://dxr.mozilla.org/mozilla-central/search?q=function%3Astart%20ext%3A.py 
<-- no results
http://dxr.mozilla.org/mozilla-central/search?q=%22def%20start%22%20ext%3A.py 
<-- results


On 10/02/2013 03:33 PM, Erik Rose wrote:

What features do you most use in MXR and DXR?

Over in the recently renamed Web Engineering group, we're working hard to 
retire MXR. It hasn't been maintained for a long time, and there's a lot of 
duplication between it and DXR, which rests upon a more modern foundation and 
has been developing like crazy. However, there are some holes we need to fill 
before we can expect you to make a Big Switch. An obvious one is indexing more 
trees: comm-central, aurora, etc. And we certainly have some bothersome UI bugs 
to squash. But I'd like to hear from you, the actual users, so it's not just me 
and Taras guessing at priorities.

What keeps you off DXR? (What are the MXR things you use constantly? Or the 
things which are seldom-used but vital?)

If you're already using DXR as part of your workflow, what could it do to make 
your work more fun?

Feel free to reply here, or attach a comment to this blog post, which talks 
about some of the things we've done recently and are considering for the future:

https://blog.mozilla.org/webdev/2013/09/30/dxr-gets-faster-hardware-vcs-integration-and-snazzier-indexing/

We'll use your input to build our priorities for Q4, so wish away!

Cheers,
Erik


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Defining a standard structured logging format for test harnesses

[Andrew Halberstadt]

tl;dr if you have opinions on a standardized structured logging format 
for test harnesses, please comment on 
https://bugzilla.mozilla.org/show_bug.cgi?id=916260



Hi all, our past intern Chris Manchester worked on implementing 
structured logging for our test harnesses this summer [1]. He wrote the 
necessary libraries and converted the xpcshell harness to a structured 
format.


We now want to convert further harnesses (like mochitest, reftest, 
marionette, etc) to start reaping the benefits of structured logging 
everywhere (there are lots of resources on the benefits of structured 
logging, but Gps' blog post last year [2] is a good place to start).


But before we do that, we want to define a standard structured logging 
format that will be used by all of our test harnesses so that consumers 
of the structured logs will have some guarantees on what to expect.


So if you have ideas about what this standard format should look like, 
please comment on bug 916260 [3]. If you have some idea of the type of 
data you'd like to get out of structured logs, feel free to comment as 
well, as this will help us make well informed decisions.


Thanks,
Andrew


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=916295
[2] 
http://gregoryszorc.com/blog/2012/12/06/thoughts-on-logging---part-1---structured-logging/

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=916260
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Pushes to Backouts on Mozilla Inbound

[Andrew Halberstadt]

On 11/05/2013 01:49 PM, Chris Peterson wrote:

On 11/5/13, 7:10 AM, James Graham wrote:

Wht data do we currently have about why the wait time is so long? If
this data doesn't exist, can we start to collect it? Are there easy wins
to be had, or do we need to think about restructuring the way that we do
builds and/or testing to achieve greater throughput?


joduinn has detailed test time stats. I believe he posted them to
dev-tree-management..?

Some of our test suites run for 90 minutes, though the original goal
AFAIU was to limit test suites to 30 minutes. Splitting long test suites
could:

1. Increase test parallelization
2. Reduce retest time when rerunning orange tests!
3. Allow some ununnecessary test cases to be excised


cpeterson


Some of the B2G emulator suites can take ~10 minutes for setup/teardown, 
so we purposefully let the chunks run a little longer to incur fewer 
penalties there. Though no one has yet done the math to determine at 
which point the benefits of chunking outweigh the cost of 
setup/teardown. I would be very interested in knowing such a formula.


Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reftests execute differently on Android or b2g?

[Andrew Halberstadt]

On 01/07/2014 07:23 PM, Neil wrote:

I tried to check in a reftest today. Apparently it fails on Android and
b2g. The failure mode appears to be that the reftest takes a screenshot
before the test has loaded (the page is still blank, whereas it should
have a red square for failure or a green square for success). Do
reftests execute differently on those platforms?


The way they are initialized is different, and they load tests from a 
server instead of a file:// url, but other than that the way they are 
executed should be the same.


Andrew

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Splitting of tests in the mochitest suite

[Andrew Halberstadt]

A downside to chunking such that skipped tests are taken into account is 
that it will make some chunks take longer to run than others. This 
negatively impacts the scheduling algorithms and ultimately increases 
infrastructure load.


The effects are especially noticeable on new platforms/applications that 
disable large swathes of tests to begin with (e.g b2g, android). For 
example, while trying to get reftests running on B2G emulators I saw 
cases where one chunk took 20 minutes while the next took ~120. I 
actually wrote a patch to do the opposite of what you are suggesting 
here (to not count skipped tests while chunking).


Correct me if I'm wrong, but I think the issue you are trying to solve 
is how to figure out what try syntax is needed to run a certain test. I 
think there are better ways to solve this problem. I.e a more expressive 
try syntax such that you can just specify the test or directory of tests 
to run and not care about what chunk it is in. Bonus points if we can 
configure mozharness to run only those tests and ignore all other tests 
that happen to be in the same chunk.


-Andrew

On 01/21/2014 12:05 PM, Ehsan Akhgari wrote:

It seems to me like the splitting algorithm for mochitests gives us
different chunking results on different platforms, see this test failure
for example: <
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=746018b05d67>.  Is this
expected?  If not, is this easy to fix so that a given test always falls
inside the same chunk if the number of chunks are equal on two different
platforms?

Thanks!
--
Ehsan




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Splitting of tests in the mochitest suite

[Andrew Halberstadt]

On 01/27/2014 03:36 PM, Ehsan Akhgari wrote:

On 1/27/2014, 7:44 AM, Andrew Halberstadt wrote:

A downside to chunking such that skipped tests are taken into
account is that it will make some chunks take longer to run than
others. This negatively impacts the scheduling algorithms and
ultimately increases infrastructure load.


I assume you are talking about the tests that are skipped in the
manifest, not on the command line, right?  In that case, yes.  But I
 should also note that using the test file count as a proxy for the
runtime requirements is... poor.  :-)


Yeah it's poor, but our tooling has a long way to go before we can start
making decisions on better information than that :)


I think there are better ways to solve this problem. I.e a more
expressive try syntax such that you can just specify the test or
directory of tests to run and not care about what chunk it is in.
Bonus points if we can configure mozharness to run only those tests
and ignore all other tests that happen to be in the same chunk.


But doesn't that mean that the code which decides what jobs to
schedule should know about what will happen as part of those jobs?  I
may be wrong, but to the best of my knowledge, that information
doesn't exist where we would want trychooser to use it.


I admit I'm not too familiar with trychooser or even buildbot for that
matter. But I do know that the scheduling step happens before the
mozharness step. And I also know that mozharness has access to the
buildbot json. So I imagine there would be a way to pass this
information along to the harness, though someone from releng could 
probably clarify this for sure.


Cheers,
Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Tech reviewer needed for FxOS reftests article

[Andrew Halberstadt]

Hey Chris, I can do a review. Though fair warning, I wrote most of it 
:). It looks like it's still accurate, though I would probably delete 
everything other than how to run them using mach. I don't think there is 
any reason to support the old manual way of doing it anymore.


Let me know what you need me to do.

Andrew

On 14/02/14 04:32 AM, Chris Mills wrote:

Hi all,

I’m currently going through all of the automation/testing articles on the 
Firefox OS zone on MDN[1], and I’m hunting for someone who knows a lot about 
Reftests, to give the FxOS reftests article[2] a review and let me know if it 
makes sense.

Let me know if you are willing to do this, and I will be eternally grateful ;-)

Chris Mills
Senior tech writer || Mozilla
developer.mozilla.org || MDN
cmi...@mozilla.com || @chrisdavidmills


[1] https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing
[2] 
https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Reftests



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Test harness options now defined in-tree -- please pull m-c before pushing to try!

[Andrew Halberstadt]

With the completion of bug 981030 test harness options are now defined 
in-tree.


This means any push without this patch [1] will fail. I already went 
ahead and landed it on all release branches, but if you have an 
integration or project branch, you may need to merge m-c in (or cherry 
pick that patch) to get things working again. If you want to push to try 
based on a rev that doesn't have it, you'll need to land it alongside 
the other changes you are pushing. Sorry for the inconvenience!


Why did I do this? There are several benefits:

A) We can easily modify the command line to the harness without needing 
to worry about affecting older branches.

B) It is now very easy to push custom command lines to try.

To push a custom command line to try, go to testing/config/mozharness, 
open the file(s) corresponding to the platform you want to change, and 
edit the '_options' key where suite is the name of the test 
harness (e.g mochitest, reftest, etc..).


Unfortunately not all test options are in-tree yet, there is still some 
work to do:
1) Talos and marionette based test options (Mn, Mnw, Gu, Gi) aren't 
in-tree yet.
2) The --this-chunk and --total-chunk options are still defined in 
buildbot for scheduling purposes. Eventually we want these in-tree too, 
but it will take some more work as these are a special case.


Let me know if you have any questions or concerns,
Andrew

[1] https://bug981030.bugzilla.mozilla.org/attachment.cgi?id=8394187
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Test harness options now defined in-tree -- please pull m-c before pushing to try!

[Andrew Halberstadt]

On 26/03/14 12:19 AM, Gregory Szorc wrote:

Andrew: I'm curious if you or anyone else has given any consideration
into making it dead simple to change the config so only a subset of
tests will be active. e.g. I'd like to save automation resources and
filter my browser chrome jobs to tests under toolkit/mozapps/extensions.
This is now possible but requires modifying N config files under
testing/config/mozharness. Could we get that down to 1 or added to Try
syntax somehow?


Releng (and I tend to agree with them) is very opposed to having one 
config affect multiple platforms (or at least multiple buildapps). Over 
time it tends to lead to messy and hard to follow configs. It also makes 
it harder in the case that you *do* only want to change a single platform.


Another more practical reason for not doing this is that often different 
platforms have vastly different command lines (e.g b2g vs firefox vs 
fennec are all different). I can't really think of an intuitive way that 
lets us share common command lines while having different sets of 
configs for the uncommon ones. IMO editing N configs isn't that big of a 
deal.. but if you really wanted to on a project branch or something, I 
suppose you could create a "global" config and have the platform 
specific ones import it. If this scheme was sane and easy to understand, 
I could possibly be convinced to land it on m-c.


Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Test harness options now defined in-tree -- please pull m-c before pushing to try!

[Andrew Halberstadt]

On 26/03/14 09:06 AM, Andrew Halberstadt wrote:

On 26/03/14 12:19 AM, Gregory Szorc wrote:

Andrew: I'm curious if you or anyone else has given any consideration
into making it dead simple to change the config so only a subset of
tests will be active. e.g. I'd like to save automation resources and
filter my browser chrome jobs to tests under toolkit/mozapps/extensions.
This is now possible but requires modifying N config files under
testing/config/mozharness. Could we get that down to 1 or added to Try
syntax somehow?


Releng (and I tend to agree with them) is very opposed to having one
config affect multiple platforms (or at least multiple buildapps). Over
time it tends to lead to messy and hard to follow configs. It also makes
it harder in the case that you *do* only want to change a single platform.

Another more practical reason for not doing this is that often different
platforms have vastly different command lines (e.g b2g vs firefox vs
fennec are all different). I can't really think of an intuitive way that
lets us share common command lines while having different sets of
configs for the uncommon ones. IMO editing N configs isn't that big of a
deal.. but if you really wanted to on a project branch or something, I
suppose you could create a "global" config and have the platform
specific ones import it. If this scheme was sane and easy to understand,
I could possibly be convinced to land it on m-c.

Andrew


To follow up, I wouldn't be opposed to having an empty global config 
that is just there to make pushing to try easier but is not used on 
production branches.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Test harness options now defined in-tree -- please pull m-c before pushing to try!

[Andrew Halberstadt]

On 26/03/14 01:45 PM, Nathan Froyd wrote:

- Original Message -

Releng (and I tend to agree with them) is very opposed to having one
config affect multiple platforms (or at least multiple buildapps). Over
time it tends to lead to messy and hard to follow configs. It also makes
it harder in the case that you *do* only want to change a single platform.


This seems like an inconsistent position, since we already have per-platform 
configs with included files for mozconfigs.  And that seems to work OK.

-Nathan


I may have misinterpreted gps' original question. I'm not opposed to 
having everything live in one file, but I am opposed to having config 
options shared across different platforms.


If the general consensus is that having something like:

mozharness_config.py:
config = {
'linux': {
'mochitest_options': [ ... ],
'reftest_options': [ ... ],
 ...
}
'windows': {
'mochitest_options': [ ... ],
'reftest_options': [ ... ],
...
}
...
}

is easier than the multi-file approach, then I'm ok with that. The 
downside to this is that if we port more configuration from mozharness 
to the tree, this file would get pretty large.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Policy for disabling tests which run on TBPL

[Andrew Halberstadt]

On 06/04/14 08:59 AM, Aryeh Gregor wrote:

On Sat, Apr 5, 2014 at 12:00 AM, Ehsan Akhgari  wrote:

Note that is only accurate to a certain point.  There are other things which
we can do to guesswork our way out of the situation for Autoland, but of
course they're resource/time intensive (basically running orange tests over
and over again, etc.)


Is there any reason in principle that we couldn't have the test runner
automatically rerun tests with known intermittent failures a few
times, and let the test pass if it passes a few times in a row after
the first fail?  This would be a much nicer option than disabling the
test entirely, and would still mean the test is mostly effective,
particularly if only specific failure messages are allowed to be
auto-retried.


Many of our test runners have that ability. But doing this implies that 
intermittents are always the fault of the test. We'd be missing whole 
classes of regressions (notably race conditions).


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Policy for disabling tests which run on TBPL

[Andrew Halberstadt]

On 04/04/14 03:44 PM, Ehsan Akhgari wrote:

On 2014-04-04, 3:12 PM, L. David Baron wrote:

Are you talking about newly-added tests, or tests that have been
passing for a long time and recently started failing?

In the latter case, the burden should fall on the regressing patch,
and the regressing patch should get backed out instead of disabling
the test.


We have no good way of identifying the regressing patch in many cases.


In cases where the regression isn't caught right away, orangefactor is 
currently the only tool at our disposal. While it is great for 
identifying priorities to fix, it is less great at drilling down through 
raw data to determine regression ranges. It is also very noisy which can 
make it hard to interpret the data.


I think we need to spend some time:
A) Exposing more raw data for tools like orangefactor to consume (e.g 
resource usage)

B) Figuring out how to separate the signal from the noise
C) Making it super easy to drill through this data to identify a 
regression range.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Policy for disabling tests which run on TBPL

[Andrew Halberstadt]

On 07/04/14 05:10 AM, James Graham wrote:

On 07/04/14 04:33, Andrew Halberstadt wrote:

On 06/04/14 08:59 AM, Aryeh Gregor wrote:

Is there any reason in principle that we couldn't have the test runner
automatically rerun tests with known intermittent failures a few
times, and let the test pass if it passes a few times in a row after
the first fail?  This would be a much nicer option than disabling the
test entirely, and would still mean the test is mostly effective,
particularly if only specific failure messages are allowed to be
auto-retried.


Many of our test runners have that ability. But doing this implies that
intermittents are always the fault of the test. We'd be missing whole
classes of regressions (notably race conditions).


In practice how effective are we at identifying bugs that lead to
instability? Is it more common that we end up disabling the test, or
marking it as "known intermittent" and learning to live with the
instability, both of which options reduce the test coverage, or is it
more common that we realise that there is a code reason for the
intermittent, and get it fixed?


I would guess the former is true in most cases. But at least there we 
have a *chance* at tracking down and fixing the failure, even if it 
takes awhile before it becomes annoying enough to prioritize. If we made 
it so intermittents never annoyed anyone, there would be even less 
motivation to fix them. Yes in theory we would still have a list of top 
failing intermittents. In practice that list will be ignored.


Case in point, desktop xpcshell does this right now. Open a log and 
ctrl-f for "Retrying tests". Most runs have a few failures that got 
retried. No one knows about these and no one looks at them. Publishing 
results somewhere easy to discover would definitely help, but I'm not 
convinced it will help much.


Doing this would also cause us to miss non-intermittent regressions, e.g 
where the ordering of tests tickles the platform the wrong way. On the 
retry, the test would get run in a completely different order and might 
show up green 100% of the time.


Either way, the problem is partly culture, partly due to not good enough 
tooling. I see where this proposal is coming from, but I think there are 
ways of tackling the problem head on. This seems kind of like a last resort.


Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Policy for disabling tests which run on TBPL

[Andrew Halberstadt]

On 07/04/14 11:49 AM, Aryeh Gregor wrote:

On Mon, Apr 7, 2014 at 6:12 PM, Ted Mielczarek  wrote:

If a bug is causing a test to fail intermittently, then that test loses
value. It still has some value in that it can catch regressions that
cause it to fail permanently, but we would not be able to catch a
regression that causes it to fail intermittently.


To some degree, yes, marking a test as expected intermittent causes it
to lose value.  If the developers who work on the relevant component
think the lost value is important enough to track down the cause of
the intermittent failure, they can do so.  That should be their
decision, not something forced on them by infrastructure issues
("everyone else will suffer if you don't find the cause for this
failure in your test").  Making known intermittent failures not turn
the tree orange doesn't stop anyone from fixing intermittent failures,
it just removes pressure from them if they decide they don't want to.
If most developers think they have more important bugs to fix, then I
don't see a problem with that.


I think this proposal would make more sense if the state of our 
infrastructure and tooling was able to handle it properly. Right now, 
automatically marking known intermittents would cause the test to lose 
*all* value. It's sad, but the only data we have about intermittents 
comes from the sheriffs manually starring them. There is also currently 
no way to mark a test KNOWN-RANDOM and automatically detect if it starts 
failing permanently. This means the failures can't be starred and become 
nearly impossible to discover, let alone diagnose.


As I mentioned in another post in this thread, we need better data and 
easier ways to drill through it. All I'm saying here is that I think 
things are probably worse than you picture them, and I think there is a 
lot of groundwork needed before it even makes sense to consider this.


-Andrew
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Hi, I want to use mach in a project that use XUL SDK without building mozilla again and again.

[Andrew Halberstadt]

On 24/06/14 09:42 PM, luoyongg...@gmail.com wrote:

I want it can generate Microsoft Visual Studio Project and without the need to 
do configure things in msys.

So that ease our development progress,
and for our own purpose, I will create my own executable and besides, the unit 
testing/auto testing infra is also needed.



Hi, you could try downloading the xulrunner sdk (and caching a local copy):
http://ftp.mozilla.org/pub/mozilla.org/xulrunner/nightly/latest-mozilla-central/

The gaia team does something similar:
https://github.com/mozilla-b2g/gaia/blob/master/Makefile#L614

You could translate that logic to a mach command/python instead of a 
Makefile.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

How many tests are disabled? (answer inside)

[Andrew Halberstadt]

Something we've been missing for a long time is high level visibility 
into the state of our tests. How many tests from suite X are we skipping 
on platform Y?


A new project called Test Informant [1] is keeping track of this data 
and allows us to generate reports with it. Here is a report generated 
from yesterday's builds:

http://people.mozilla.org/~ahalberstadt/informant-reports/daily/2014-09-29.informant-report.html

For more details including known limitations and future work, see my 
blog post on the topic:

http://ahal.ca/blog/2014/how-many-tests-are-disabled/

I intend to automate the process of generating both daily and weekly 
reports (daily reports will show how many tests were enabled/disabled 
from one day to the next, weekly reports from one week to the next). I'd 
like to post an automated weekly report to dev.platform starting next week.


Please let me know if this sounds reasonable, if you have 
feedback/questions or if you notice any inconsistencies with the report 
and the real world state of manifests.


Cheers,
Andrew

[1] https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: How many tests are disabled? (answer inside)

[Andrew Halberstadt]

On 30/09/14 11:19 AM, jmath...@mozilla.com wrote:

Is there any chance you could generate a report on tests disabled in e10s mode? 
This would be very useful for the e10s team as we currently have a lot of tests 
disabled which need to get fixed.



It should be do-able, I'll look into it. I think having a way to set 
arbitrary filter values in the configuration could help us in other 
scenarios as well.


One thing to note though is that there wouldn't be a way to keep it 
in-sync with m-c. So if the name of the variable changes, or gets 
removed or something, we'd have to update the test-informant 
configuration manually.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: How many tests are disabled? (answer inside)

[Andrew Halberstadt]

On 30/09/14 03:12 PM, Chris Peterson wrote:

Is there a stable URL that always points to the most recent report? I
recommend creating a symlink called something like
"informant-report.html" or "latest.informant-report.html" to the most
recent report. ("index.html" would be a bad name because it prevent
directory browsing for people who might want to poke around for an older
report.)

chris


Not yet, but good idea. My people account is just a temporary place to 
store the reports until the ACL's between the test-informant server and 
brasstacks.mozilla.com get opened.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: How many tests are disabled? (answer inside)

[Andrew Halberstadt]

latest.informant-report.html symlink added!

On 30/09/14 03:12 PM, Chris Peterson wrote:

On 9/30/14 7:35 AM, Andrew Halberstadt wrote:

A new project called Test Informant [1] is keeping track of this data
and allows us to generate reports with it. Here is a report generated
from yesterday's builds:
http://people.mozilla.org/~ahalberstadt/informant-reports/daily/2014-09-29.informant-report.html



Is there a stable URL that always points to the most recent report? I
recommend creating a symlink called something like
"informant-report.html" or "latest.informant-report.html" to the most
recent report. ("index.html" would be a bad name because it prevent
directory browsing for people who might want to poke around for an older
report.)

chris




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Test Informant Report - Week of 2014-09-28

[Andrew Halberstadt]

See:
http://people.mozilla.org/~ahalberstadt/informant-reports/weekly/2014-10-06.informant-report.html

Hi all, last week I made a post saying to expect to see more of these 
reports. This is the first of a series of weekly reports I intend to 
post to dev.platform outlining which tests have been enabled or disabled 
over the course of the week.


Since last week, I've added:
* Top level summaries of enabled/disabled tests to each suite/platform
* mochitest e10s support
* permanent urls (note: I'll be moving the reports off of my people 
account in the near future)


This process will eventually be fully automated, but is blocking on some 
permissions being set on the web server.


Let me know if you have any questions or concerns!
Andrew

p.s You may notice a lot of mochitests being enabled in today's daily 
report, they weren't actually ever disabled. Test Informant just thought 
they were because the mochitest harness was setting some manifest 
filters at runtime. The discrepancy has now been fixed.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Test Informant Report - Week of 2014-10-05

[Andrew Halberstadt]

Test Informant report for 2014-10-12.

State of test manifests at revision f547cf19d104.
Using revision 9ee9e193fc48 as a baseline for comparisons.
Showing tests enabled or disabled between 2014-10-06 and 2014-10-12.

79% of tests across all suites and configurations are enabled.


Summary
---
marionette- ↑0↓1600 - 93%
mochitest-a11y- ↑0↓0   - 99%
mochitest-browser-chrome  - ↑72↓0  - 95%
mochitest-browser-chrome-e10s - ↑16↓0  - 29%
mochitest-chrome  - ↑10↓0  - 97%
mochitest-plain   - ↑3456↓0 - 75%
mochitest-plain-e10s  - ↑28↓136 - 66%
xpcshell  - ↑988↓0 - 92%


Full Report
---
http://people.mozilla.org/~ahalberstadt/informant-reports/weekly/2014-10-12.informant-report.html


Notes
-
* Marionette webapi tests were wrongfully counted as being run on 
desktop platforms (now fixed, ignore the 1600 disabled tests there)

* android-opt now being tracked under mochitest-plain and xpcshell
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Test Informant Report - Week of 2014-10-05

[Andrew Halberstadt]

On 14/10/14 11:58 AM, Bill McCloskey wrote:

On Tue, Oct 14, 2014 at 8:45 AM, Dave Townsend  wrote:

Is there any way to see the breakdown of which testsgot enabled. I'm
surprised to see mochitest-plain rise by 3456 yet mochitest-plain-e10s rise
by only 28. I'd love to find out which tests got turned on on one but not
the other.


If you go to the full report link in the original email, you can click
on each line to get deltas. The 3456 enabled mochitest-plain tests
look like mostly Android tests.

-Bill


Yep, we just started collecting data for Android last week, so in the 
eyes of the test-informant every android mochitest was "added". The fact 
that this shows up in the summary is a bug that I should probably fix..


Andrew


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform