Epic stuff Sean, thanks for sharing! It's always gratifying when
refactoring work like this uncovers issues that had previously gone
unnoticed.
Cheers,
Ryan
On 24 March 2017 at 05:51, Sean McArthur <[email protected]> wrote:
> (tl;dr is at the bottom)
>
> Some background: our (auth-server) remote tests have always run in such a
> way that each file full of remote tests starts up several child processes to
> run an auth-server and other server helper stubs. The tests in the file
> would make requests against this new flock of children, and then shut them
> down, before moving on to the next test file.
>
> Some of the problems we would encounter because of this setup, by my own
> rating of importance:
>
> - Since the tests took a long time to run (around 4 minutes), I'd find
> myself trying as much as possible to not run them. Our CI would run them, so
> they weren't ever forgotten, but a developer being passively discouraged to
> run tests is not a great situation.
>
> - We would get sporadic test failures and time outs due to child process
> startup woes, and not being able to reflect exceptions in the child
> processes back to the test runner. It's like screaming children in the line
> at Disney.
>
> - Quite often, the child processes wouldn't be killed when the parent test
> runner process quit, and then we'd have odd errors when consecutive runs
> tried to start up processes to listen on an already used port. It's a hard
> knock life, indeed.
>
> - Running the tests was slow, even on CI, so waiting for the green
> checkmarks meant you needed to go do something else and remember to come
> back and merge later. (Unfortunately, a lot of the time is spent in `npm
> install`, so it still takes longer than I'd like.)
>
> - Even though we hoped different processes would isolate the tests, we
> polluted the process `ENV` to customize the children, which had an affect of
> causing test order to affect whether tests passed in some cases.
>
> Pull Request #1760 is shutting down the orphanage. It includes some
> adjustments to the `TestServer` type used in our remote tests. It no longer
> spawns a bunch of children, but instead just starts up all the servers in
> the same process. This fixes (or greatly reduces) each of the problems
> listed above. We now have the full `npm test` run executing in under 1
> minute! And there are no more dangling orphans left behind at fire stations.
>
> Fun little bonus, this even discovered a couple of exceptions that were just
> silently ignored, due to how Hapi reports them and how we ignored that log
> output. Thankfully, mostly harmless.
>
> tl;dr - Running the auth-server tests now starts all needed servers in the
> same process, improving correctness, reducing coupling between tests, and
> improving test run speed by around 4x!
>
> _______________________________________________
> Dev-fxacct mailing list
> [email protected]
> https://mail.mozilla.org/listinfo/dev-fxacct
>
_______________________________________________
Dev-fxacct mailing list
[email protected]
https://mail.mozilla.org/listinfo/dev-fxacct
