On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefa...@gmail.com> wrote: > > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote: > > 3) The community will be given all devised performance measurement methods > > in the form of easily reproducible step-by-step setup and execution > > procedures. > > Tracking performance is a good idea and something that has not been done > upstream yet.
Thanks for the interest, Stefan! > A few questions: > > * Will benchmarks be run automatically (e.g. nightly or weekly) on > someone's hardware or does every TCG architecture maintainer need to > run them manually for themselves? If the community wants it, definitely yes. Once the methodology is developed, it should be straightforward to setup nightly and/or weekly benchmarks - that could definitely include sending mails with reports to the entire list or just individuals or subgroups. The recipient choice is just a matter or having decent criteria about appropriateness of information within the message (e.g. not to flood the list with the data most people are not really interested). For linux-user tests, they are typically very quick, and nightly tests are quite feasible to run. On someone hardware, of course, and consistently always on the same hardware, if possible. If it makes sense, one could setup multiple test beds with a variety of hardware setups. For system mode tests, I knoe they are much more difficult to automate, and, on top of that, there could be greater risk of hangs/crashes Also, considering the number of machines we support, those tests could consume much more time - perhaps even one day would not be sufficient, if we have many machines and boot/shutdown variants. For these reason, perhaps weekly executions would be more appropriate for them, and, in general, given greater complexity, the expectation from system-mode performance tests should be better kept quite low for now. > * Where will the benchmark result history be stored? > If emailing is set up, the results could be reconstructed from emails. But, yes, it would be better if the result history is kept somewhere on an internet-connected file server Yours, Aleksandar > Stefan
