On Wednesday 2015-11-04 12:46 -0500, William Lachance wrote:
> On 2015-11-04 10:55 AM, William Lachance wrote:
> >
> >1. Relatively deterministic.
> >2. Something people actually care about and are willing to act on, on a
> >per-commit basis. If you're only going to look at it once a quarter or
> >so, it doesn't need to be in Perfherder.
> >
> >Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
> >add stuff incrementally, to make sure Perfherder can handle the load,
> >but I'd love to hear all your ideas.
> 
> Someone mentioned "test times" to me in private email.

That was me.  (I didn't feel like sending a late-at-night
one-sentence email to the whole list, and figured there was a decent
chance that somebody else would mention it as well.)

I think they're worth tracking because we've had substantial
performance regressions (I think including as bad as a doubling in
times) that weren't caught quickly, and led to substantially worse
load on our testing infrastructure.

> I do think test times are worth tracking, but probably not in Perfherder:
> test times might not be deterministic depending on where / how they're
> running (which makes it difficult to automatically detect regressions and
> sheriff them on a per commit basis) and regardless there's too much data to
> really be manageable by Perfherder's intended interface even if that problem
> were magically solved.

It seems like if we're running the same tests on different sorts of
machines, we could track different perf numbers for the test run on
different machine classes.

We'd also want to measure the test time and *not* the time spent
downloading the build.

And we'd probably want to measure the total time across chunks so
that we don't count redistribution between chunks as a set of
regressions and improvements.

So that does make it a bit difficult, but it does seem doable.

> As a possible alternative, I believe Kyle Lahnakoski's ActiveData project
> (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData) already *does*
> track this type of data but last I heard he was looking for more feedback on
> how to alert/present it to the platform community. If you have any ideas on
> this, please let him know (he's CC'ed). :)

Perhaps, but I have no idea how to use it or what that would look
like.  The wiki page explicitly says it's for automated clients and
not by humans; it would be useful to see an example of such an
automated client to have an idea of how this would work.

-David

-- 
𝄞   L. David Baron                         http://dbaron.org/   𝄂
𝄢   Mozilla                          https://www.mozilla.org/   𝄂
             Before I built a wall I'd ask to know
             What I was walling in or walling out,
             And to whom I was like to give offense.
               - Robert Frost, Mending Wall (1914)

Attachment: signature.asc
Description: Digital signature

_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reply via email to