On Tue, Jul 15, 2014 at 12:15 PM, Ted Mielczarek <t...@mielczarek.org> wrote:
> On 7/15/2014 2:49 PM, Dave Townsend wrote: > > Since forever Jetpack tests in the Firefox trees have been run using our > > custom python CFX tool which is based on a fork of an ancient version of > > mozrunner. This causes us a number of problems. Keeping up with tree > > visibility rules is hard. Some features from newer versions of mozrunner > > like crash stack handling aren't available and our attempts to update to > > the newer mozbase have been blocked on trying to get some of our forked > > code accepted. It also makes it hard for Mozilla other developers to run > > our tests as CFX has a very different syntax to the other test suites. > > > > We've started investigating switching away from CFX and instead using the > > python automation that the mochitests use. This would work somewhat > > similarly to browser-chrome tests, runtests.py will startup Firefox and > > overlay some XUL and JS on the main window from where we can run the > > existing JS parts of the Jetpack test suites. > > > > There are many benefits here. The runtests.py code is well used and known > > to be resilient. It supports things like screenshots on failures and > crash > > stacks that Jetpack tests don't currently handle. We'll use manifest > files > > like the other test suites so disabling tests per platform will be easy. > > Excellent mach integration will make running individual tests simple. It > > also makes it possible to use commonjs style tests elsewhere in the tree. > > Release engineering should find managing the Jetpack tests a lot easier > as > > they behave just like other mochitests. > > > > My initial experiment last week shows that this will work. The first part > > of our tests (package tests) is running and passing on my local machine > and > > I expect to have the add-on tests working this week. > > > > I wanted to give everyone a heads up about this work to give you all a > > chance to ask questions or raise objections. The changes to runtests and > > the build system are minimal, just adding support for new manifest types > > really but I will be needing reviews for those. We'll also have to make > the > > buildbot changes to switch over to use these new tests but I expect that > to > > be pretty straightforward. > > _______________________________________________ > I am totally into the sentiment here, but I'm not sure that I'm into > your exact plan of action. The Jetpack tests definitely have a lot of > issues due to not using the same infrastructure as other test harnesses, > so I'd be glad to see that get fixed. On the other hand, the Mochitest > harness is already unduly complicated by trying to do too many > things--plain Mochitests, chrome Mochitests, browser-chrome tests and > all the other variants are all shoehorned in there and it's a mess. I'd > rather not shoehorn yet another test framework into the Mochitest > umbrella. However! In the modern era Mochitest is mostly comprised of > some glue code that uses mozbase modules to do all the hard work. I > think you can cobble together a reasonable facsimile that runs the > Jetpack tests in a cleaner fashion by using the mozbase modules directly. > Creating a new test harness (even one based on top of mozbase) loses us some of the benefits of this. By re-using existing code we get improvements to that code for free. Right now making changes to CFX is hard in part because the people who wrote the code have moved on and almost no-one on the team really knows python. For example when new test requirements come upand are fixed in whatever harness we switch to > > Alternately, as gps pointed out in his reply, you might want to look at > Marionette as a starting point. The Marionette test harness uses mozbase > as well, but gives you the power of the Marionette protocol to control > the browser. If we were writing the Mochitest harness from scratch today > I would base it on top of Marionette. > I'm not set on the mochitest harness or any harness really. The mochitest harness seemed the simplest because I know it reasonably well and browser-chrome already does exactly what we need. Ignoring the work to support the manifests for these tests it took maybe 30 minutes to get the mochitest harness supporting jetpack package tests. I'll take a look at Marionette and see what I can learn. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
