If this only involves tiny changes to mochitest and it's ready, I'd go
ahead and do that.
I am interested in seeing what your requirements are, though, and
figuring out if we could meet them later with a better architected
solution, whether it's Marionette or something else. Mochitest is kind
of a monster, and the more we hack on corner cases the more fragile and
unmaintainable it becomes.
Jonathan
On 8/5/2014 9:20 AM, Dave Townsend wrote:
On Mon, Aug 4, 2014 at 6:21 PM, Gregory Szorc <g...@mozilla.com> wrote:
On 8/4/14, 10:39 AM, Dave Townsend wrote:
I've done a little investigation into marionette and I've found a few
issues with it:
Firstly it doesn't look like running marionette directly or through mach
allows developers to select individual directories or test files to run,
rather it is a one-shot affair. This is very inconvenient for development.
Secondly marionette doesn't seem to be built to scale to many test
types. It uses regular expressions on filenames to determine the test
type, as it happens the Jetpack tests do use a different form to the
existing marionette tests so it's not out of the question but still
makes me wary of adding a new test type.
Thirdly I can't run marionette tests locally, they consistently fail
quite badly.
These problems make marionette a less than desirable option for use as a
base for our test harness right now so I plan to get my work to make
mochitests run jetpack tests completed this week and submit it for
review. If Marionette becomes a better choice in the future a lot of the
work I'm doing right now carries over, it will be simpler to switch from
mochitest to marionette than it is switching to mochitest.
The issues listed seem fixable. I would rather we spend energy improving
Marionette than piling yet more things on top of mochitest's haphazard base.
The various automation "failures" in the past few weeks should be reason
enough to avoid mochitest and go with a better-engineered and tested
solution (marionette).
Who is going to do that work? I have patches that vastly improve the
testing situation for jetpack tests by allowing other developers run them
more easily, making them easier for releng to manage and most importantly
making them meet tbpl visibility requirements. They involve tiny changes to
the mochitest harness code. We're already hidden by default on tinderbox
and are hitting problems because of it, I'd rather go ahead and finish up
this work than wait for some future time when marionette can be upgraded to
meet our requirements.
_______________________________________________
tools mailing list
to...@lists.mozilla.org
https://lists.mozilla.org/listinfo/tools
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
