On 01/04/2019 23:13, Steve Fink wrote:
On 4/1/19 11:36 AM, Brian Grinstead wrote:
Based on my own experience and discussions with others, the workflow
for adding new mochitests isn't great. Commonly, it looks like:
"copy/paste a test in the same directory, add the new test to the
relevant manifest file, empty out the actual test bits, write your
test". In my experience this is prone to issues like forgetting to add
the new test to the manifest, or not fully replacing boilerplate like
bug numbers from the copied test.
There's a script in tree I was unaware of until last week called
gen_template.pl that's intended to help here, but it does leave a few
issues open:
1) It doesn't help with finding the manifest file and adding the new
test to it.
2) The boilerplate it generates is outdated (for example, it sets
type="application/javascript" even in HTML documents, it doesn't
include add_task, etc).
3) It supports only mochitest-chrome and mochitest-plain.
Last week I prototyped a new mach command to fix (1) and (2), and
expand (3) to include browser-chrome mochitests. If it's helpful, it
could be extended to more test types as well. When you run the command
it will create a file with the appropriate boilerplate and add it to
the manifest file (chrome.ini, mochitest.ini, browser.ini depending on
the type). This way you can immediately run the test with `./mach
mochitest`.
It sounds great to me, but I'm wondering if the generic name is
intentional or not. Various groups within Mozilla assume different
things by 'test'. Is`mach addtest` intended to only be for mochitests?
If so, then perhaps `mach addmochitest` is a better name, even it's a
bit of mouthful. My reasoning is that there's already a distinction
between `mach mochitest` and `mach test`, where the latter attempts to
be general and support a bunch of different kinds of tests. Having `mach
test` assume mochitests would be highly confusing to me, at least.
(Though I'm not sure that `mach test` really works; it seems like I
usually have to run the more specific command.)
I'm also pretty worried by this. For web-platform features ensuring
interop is critical and as such web-platform-tests should be preferred
over mochitests where possible. But every time we build features with a
mochitest-first approach it undermines that.
For web-platform-tests we already have ./mach wpt-create, so I think we
should either roll that functionality in to the new command as part of
the initial featureset or have one command per supported test type (i.e.
call this mach mochitest-create).
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
