I've done a little investigation into marionette and I've found a few issues with it:
Firstly it doesn't look like running marionette directly or through mach allows developers to select individual directories or test files to run, rather it is a one-shot affair. This is very inconvenient for development. Secondly marionette doesn't seem to be built to scale to many test types. It uses regular expressions on filenames to determine the test type, as it happens the Jetpack tests do use a different form to the existing marionette tests so it's not out of the question but still makes me wary of adding a new test type. Thirdly I can't run marionette tests locally, they consistently fail quite badly. These problems make marionette a less than desirable option for use as a base for our test harness right now so I plan to get my work to make mochitests run jetpack tests completed this week and submit it for review. If Marionette becomes a better choice in the future a lot of the work I'm doing right now carries over, it will be simpler to switch from mochitest to marionette than it is switching to mochitest. On Tue, Jul 15, 2014 at 10:56 PM, Henrik Skupin <hsku...@mozilla.com> wrote: > Gregory Szorc wrote on 07/15/2014 09:04 PM: > > > On 7/15/14, 11:49 AM, Dave Townsend wrote: > >> Since forever Jetpack tests in the Firefox trees have been run using our > >> custom python CFX tool which is based on a fork of an ancient version of > >> mozrunner. This causes us a number of problems. Keeping up with tree > >> visibility rules is hard. Some features from newer versions of mozrunner > >> like crash stack handling aren't available and our attempts to update to > >> the newer mozbase have been blocked on trying to get some of our forked > >> code accepted. It also makes it hard for Mozilla other developers to run > >> our tests as CFX has a very different syntax to the other test suites. > >> > >> We've started investigating switching away from CFX and instead using > the > >> python automation that the mochitests use. This would work somewhat > >> similarly to browser-chrome tests, runtests.py will startup Firefox and > >> overlay some XUL and JS on the main window from where we can run the > >> existing JS parts of the Jetpack test suites. > >> > >> There are many benefits here. The runtests.py code is well used and > known > >> to be resilient. It supports things like screenshots on failures and > crash > >> stacks that Jetpack tests don't currently handle. We'll use manifest > files > >> like the other test suites so disabling tests per platform will be easy. > >> Excellent mach integration will make running individual tests simple. It > >> also makes it possible to use commonjs style tests elsewhere in the > tree. > >> Release engineering should find managing the Jetpack tests a lot easier > as > >> they behave just like other mochitests. > >> > >> My initial experiment last week shows that this will work. The first > part > >> of our tests (package tests) is running and passing on my local machine > and > >> I expect to have the add-on tests working this week. > >> > >> I wanted to give everyone a heads up about this work to give you all a > >> chance to ask questions or raise objections. The changes to runtests and > >> the build system are minimal, just adding support for new manifest types > >> really but I will be needing reviews for those. We'll also have to make > the > >> buildbot changes to switch over to use these new tests but I expect > that to > >> be pretty straightforward. > > > > Was Marionette considered? From what little I know (jgriffin and others > > can correct me), Marionette seems like the logical base for this test > suite. > > Adding the tools mailing list, so that members of the A-team are aware > of this thread, and can answer appropriately. > > -- > Henrik Skupin > Senior Test Engineer > Mozilla Corporation > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
