The main problem is that the current syntax is very inflexible. And tags
are just one of many things we want to make easier when pushing to try
(think of things like running directories or individual tests and all
the debug options like --run-until-failure). A try syntax that knows how
to handle all those cases will be just as convoluted as editing the
mozharness configs.
Also, how things are currently set up, we'd need to get the try syntax
to somehow modify how buildbot runs tests. This would be difficult if
not impossible. Taskcluster may make this easier, but I wouldn't count
on it..
On 31/03/15 02:12 PM, Bobby Holley wrote:
Is there any reason we couldn't put it in the try syntax instead?
On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
ahalberst...@mozilla.com> wrote:
It's technically already possible by modifying the in-tree mozharness
configs here:
https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
However it's not easy to figure out what needs to be modified to get the
desired results. Some sort of |mach try| like command is going to be worked
on in Q2 to make pushing various test configurations to try easier, I'm not
100% sure how it will be implemented yet though.
In the meantime, if you get stuck trying to modify the mozharness configs,
let me know and I can come up with an example patch.
-Andrew
On 31/03/15 01:06 PM, Bobby Holley wrote:
This sounds awesome! Is there an estimate of when we'll be able to use it
for try pushes?
On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
ahalberst...@mozilla.com> wrote:
As of bug 987360, you can now run all tests with a given tag for
mochitest (and variants), xpcshell and marionette based harnesses. Tags can
be applied to either individual tests, or the DEFAULT section in manifests.
Tests can have multiple tags, in which case they should be comma delimited.
To run all tests with a given tag, pass in --tag <tag name> to the mach
command.
For example, let's say we want to group all mochitest-plain tests related
to canvas together. First we'd add a 'canvas' tag to the DEFAULT section in
https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
[DEFAULT]
tags = canvas
We notice there is also a canvas related test under dom/media, namely:
https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
Let's pretend it is already tagged with the 'media' tag, but that's ok,
we can add a second tag no problem:
[test_video_to_canvas.html]
tags = media,canvas
Repeat above for any other tests or manifests scattered in the tree that
are related to canvas. Now we can run all mochitest-plain tests with:
./mach mochitest-plain --tag canvas
You can also run the union of two tags by specifying --tag more than once
(though the intersection of two tags is not supported):
./mach mochitest-plain --tag canvas --tag media
So far the xpcshell (./mach xpcshell-test --tag name) and marionette
(./mach marionette-test --tag name) commands are also supported. Reftest is
not supported as it has its own special manifest format.
Applying tags to tests will not affect automation or other people's tags.
So each organization or team should feel free to use tags in whatever
creative ways they see fit. Eventually, we'll start using tags as a
foundation for some more advanced features and analysis. For example, we
may implement a way to run all tests with a given tag across multiple
different suites.
If you have any questions or things aren't working, please let me know!
Cheers,
Andrew
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
