On 8/20/2014 11:24 AM, Gregory Szorc wrote:
It sounds like the "copy build to remote machine so we can run tests
there" is somewhat common and needs to be better supported by the tools.
I think it makes sense to leverage test archives and build packages
for this. Since mozharness already supports running things from there,
I guess the missing piece is some glue code that invokes mozharness
with a sane set of default options. That missing piece sounds a lot
like what mach does. We talked about bundling mach in the tests
archive to facilitate things like this. Perhaps we should move forward
with that? Or is there a better way?
The problem is at the unholy intersection of build, test, and
automation responsibilities. Who's the owner? A-Team?
Yes, I think we should own this. I think the ultimate solution involves
several pieces which probably look something like this:
- bundling mach in tests.zip and making it operable from there (I just
filed https://bugzilla.mozilla.org/show_bug.cgi?id=1058923 to track this)
- making mozharness scripts invoke mach where possible, reducing the
complexity of these scripts and providing a common invocation path
- adding some more intelligent tooling to mozharness to make it easier
to operate with local builds and test packages
- add self tests for our harnesses to catch problems in
developer-friendly code paths that aren't used in automation (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1048884)
- add mach targets for triggering try jobs using local builds and/or
test packages
With this set of tasks, I think we can meet a set of requirements that
will prevent a lot of developer frustration:
- mach targets/developer-friendly harness options shouldn't get broken
even though tests pass in automation
- it should be trivially easy to run tests using the same options used
in buildbot
- it should be trivially easy to run tests from tests.zip
- it should be trivially easy to use local test packages and builds in
try jobs
I'll follow-up with additional posts as we flesh out a plan to tackle
this in more detail.
Jonathan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
