- Original Message -
> Talos memory measurements aren't very good because it cycles through
> multiple sites in a single tab. So it sometimes catches start-up
> memory consumption regressions (Firefox Health Report was a recent
> case) but it doesn't get much beyond that.
Another problem
Writing a lot of performance tests creates the problem that those tests
will take a long time to run. The nature of performance tests is that each
test must run for a relatively long time to get meaningful results.
Therefore I doubt writing lots of different performance tests can scale.
(Maybe w
On Tue, Mar 5, 2013 at 11:47 AM, Dave Mandelin wrote:
>
> It appears that there a few areas that are only covered by Talos for now,
> though. I think in that category we have warm startup time via Ts, and basic
> layout performance via Tp. I'm not sure about memory, because we do seem to
> dete
On Monday, March 4, 2013 5:17:29 PM UTC-8, Gregory Szorc wrote:
> On 3/4/13 5:09 PM, Dave Mandelin wrote:
>
> >> We already don't back back out changes for regressing a benchmark like
> >> we back them out for regressing tests. I think this is at least
> >> partially because a general sentiment
On Monday, March 4, 2013 4:56:46 PM UTC-8, Jeff Hammel wrote:
> I'll point out and really this is about all I have to say on this thread
> that while perf testing (that is, recording results) may bewell, not
> easy, but not too awful that rigorous analysis of what the data means
> and if the
On 3/4/13 5:09 PM, Dave Mandelin wrote:
We already don't back back out changes for regressing a benchmark like
we back them out for regressing tests. I think this is at least
partially because a general sentiment that not all of our benchmarks
correlate strongly to what they're trying to measur
> We already don't back back out changes for regressing a benchmark like
> we back them out for regressing tests. I think this is at least
> partially because a general sentiment that not all of our benchmarks
> correlate strongly to what they're trying to measure.
I know this has been a hot top
On Monday, March 4, 2013 5:42:39 AM UTC-8, Ed Morley wrote:
> (CCing auto-to...@mozilla.com)
>
> jmaher and jhammel will be able to comment more on the talos specifics,
> but few thoughts off the top of my head:
>
> It seems like we're conflating multiple issues here:
> 1) "[talos] is a separa
I'll point out and really this is about all I have to say on this thread
that while perf testing (that is, recording results) may bewell, not
easy, but not too awful that rigorous analysis of what the data means
and if there is a regression is often hard since it is often the case,
as evide
Writing a lot of performance tests creates the problem that those tests
will take a long time to run. The nature of performance tests is that each
test must run for a relatively long time to get meaningful results.
Therefore I doubt writing lots of different performance tests can scale.
(Maybe we c
On Monday, March 4, 2013 5:15:56 AM UTC-8, Jim Mathies wrote:
> For metrofx we’ve been working on getting omtc and apzc running in the
> browser. One of the things we need to be able to do is run performance tests
> that tell us whether or not the work we’re doing is having a positive effect
> o
On 3/4/13 9:36 AM, Gregory Szorc wrote:
If all your tests are declared the same way, then presumably the test
running code would be similar and capturing performance data would
require a single implementation affecting all test suites instead of N
1-off solutions.
We've talked about this befor
> 1) something checked into mc anyone can easily author or run (for tracking
> down regressions) without having to checkout a separate repo, or setup and
> run a custom perf test framework.
I don't oppose the gist of what you're suggesting here, but please
keep in mind that small perf changes ar
"Boris Zbarsky" wrote in message
news:...
> On 3/4/13 8:15 AM, Jim Mathies wrote:
> > So to work around this I’ve been putting together some basic perf tests I
> > can use to measure performance using the mochitest framework.
>
> How are you dealing with the fact that mochitest runs on heteroge
On 3/4/13 5:15 AM, Jim Mathies wrote:
For metrofx we’ve been working on getting omtc and apzc running in the browser.
One of the things we need to be able to do is run performance tests that tell
us whether or not the work we’re doing is having a positive effect on perf. We
currently don’t hav
On 3/4/13 8:15 AM, Jim Mathies wrote:
So to work around this I’ve been putting together some basic perf tests I can
use to measure performance using the mochitest framework.
How are you dealing with the fact that mochitest runs on heterogeneous
hardware (including VMs and the like last I chec
Good points, comments below.
"Ed Morley" wrote in message
news:...
(CCing auto-to...@mozilla.com)
jmaher and jhammel will be able to comment more on the talos specifics,
but few thoughts off the top of my head:
It seems like we're conflating multiple issues here:
1) "[talos] is a separat
-Joel
- Original Message -
From: "Ed Morley"
To: "Jim Mathies" , auto-to...@mozilla.com
Cc: dev-platform@lists.mozilla.org
Sent: Monday, March 4, 2013 8:42:39 AM
Subject: Re: proposal: replace talos with inline tests
(CCing auto-to...@mozilla.com)
jmaher and jhammel will b
(CCing auto-to...@mozilla.com)
jmaher and jhammel will be able to comment more on the talos specifics,
but few thoughts off the top of my head:
It seems like we're conflating multiple issues here:
1) "[talos] is a separate repo from mc"
2) "[it's a hassle to] test the test to be sure it’s wo
For metrofx we’ve been working on getting omtc and apzc running in the browser.
One of the things we need to be able to do is run performance tests that tell
us whether or not the work we’re doing is having a positive effect on perf. We
currently don’t have automated tests up and running for met
20 matches
Mail list logo
