> On 12. Dec 2019, at 21:51, Paul Davis <paul.joseph.da...@gmail.com> wrote:
>
> Hey all,
>
> I was poking around at Jenkins the other day trying to get a good idea
> of how much time we're spending in various parts of the build. It
> occurred to me that one good way to at least investigate our eunit
> test suite is to parse all of the generated surefire reports. I spent
> an hour or so yesterday throwing a python script together to see what
> that might look like.
>
> I've posted an example output and the script itself here:
>
> https://gist.github.com/davisp/7064c1ef0dc94a99c739729b97fef10e
>
> So far it shows some aggregate timing information along with the top
> few results for total suite time, total setup/teardown time, and
> longest individual test times. Not included in the output, but any
> test failures or skipped tests are also included in the output which
> has the nice benefit of showing exactly which test or tests failed
> during a run.
>
> So far a few interesting things jump out. First, we spend more time in
> setup/teardown fixtures than we do running actual tests which I found
> to be rather surprising. Not so surprising is that our property tests
> are taking the longest for each run.
>
> I thought it might be interesting to include a version of this script
> (with perhaps a somewhat more succinct output) at the end of every
> eunit run (or perhaps failed eunit run) to list out the errors for
> easier debugging of failed CI runs. And then beyond that, having it in
> the source tree would allow devs to poke around and trying to speed up
> some of the slower tests.
>
> Another thought was to take a look at moving the property based tests
> to a nightly build and then actually increasing their run times to
> cover more of the state space for each of those tests. That way we'd
> be doing a bit more of the normal property based testing where its
> more about long soaks to find edge cases rather than acceptance
> testing for a particular change.
I like this idea.
Best
Jan
>
> So far I've just thrown this together. If there's enough of a
> consensus that we'd like to see something of this nature I can work a
> bit more on improving the script to have something that's useful both
> locally for reducing the test suite times and also for spotting errors
> that failed a CI run.
>
> Paul
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
