The A-Team is embarking on a project to improve the developer experience when running unittests locally. This project will address the following frequently-heard complaints:
* Locally developers often use mach to run tests, but tests in CI use mozharness, which can result in different behaviors. * It's hard to reproduce a try job because it's hard to set up the test environment and difficult to figure out which command-line arguments to use. * It's difficult to run tests from a tests.zip package if you don't have a build on that machine and thus can't use mach. * It's difficult to run tests through a debugger using a downloaded build. The quintessential use case here is making it easy to reproduce a try run locally, without a local build, using a syntax something like: * runtests --try 2844bc3a9227 Ideally, this would download the appropriate build and tests.zip package, bootstrap the test environment, and run the tests using the exact same arguments as are used on try, optionally running it through an appropriate debugger. You would be able to substitute a local build and/or local tests.zip package if desired. You would be able to override command-line arguments used in CI if you wanted to, otherwise the tests would be run using the same args as in CI. What other use cases would you like us to address, which aren't derivatives of the above issues? Thanks for your input, Jonathan _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
