On Sun, Jan 26, 2020 at 05:50:24PM +0100, Aleksandar Markovic wrote: > On Wed, Jan 22, 2020 at 12:28 PM Stefan Hajnoczi <stefa...@gmail.com> wrote: > > > > On Tue, Jan 21, 2020 at 03:07:53PM +0100, Aleksandar Markovic wrote: > > > On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefa...@gmail.com> > > > wrote: > > > > > > > > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote: > > > > > 3) The community will be given all devised performance measurement > > > > > methods in the form of easily reproducible step-by-step setup and > > > > > execution procedures. > > > > > > > > Tracking performance is a good idea and something that has not been done > > > > upstream yet. > > > > > > Thanks for the interest, Stefan! > > > > > > > A few questions: > > > > > > > > * Will benchmarks be run automatically (e.g. nightly or weekly) on > > > > someone's hardware or does every TCG architecture maintainer need to > > > > run them manually for themselves? > > > > > > If the community wants it, definitely yes. Once the methodology is > > > developed, it should be straightforward to setup nightly and/or weekly > > > benchmarks - that could definitely include sending mails with reports > > > to the entire list or just individuals or subgroups. The recipient > > > choice is just a matter or having decent criteria about > > > appropriateness of information within the message (e.g. not to flood > > > the list with the data most people are not really interested). > > > > > > For linux-user tests, they are typically very quick, and nightly tests > > > are quite feasible to run. On someone hardware, of course, and > > > consistently always on the same hardware, if possible. If it makes > > > sense, one could setup multiple test beds with a variety of hardware > > > setups. > > > > > > For system mode tests, I knoe they are much more difficult to > > > automate, and, on top of that, there could be greater risk of > > > hangs/crashes Also, considering the number of machines we support, > > > those tests could consume much more time - perhaps even one day would > > > not be sufficient, if we have many machines and boot/shutdown > > > variants. For these reason, perhaps weekly executions would be more > > > appropriate for them, and, in general, given greater complexity, the > > > expectation from system-mode performance tests should be better kept > > > quite low for now. > > > > > > > * Where will the benchmark result history be stored? > > > > > > > > > > If emailing is set up, the results could be reconstructed from emails. > > > But, yes, it would be better if the result history is kept somewhere > > > on an internet-connected file server > > > > Thanks. I don't want to overcomplicate this project. The main thing is > > to identify the stakeholders (TCG target maintainers?) and make sure > > they are happy. > > > > Yes, Stefan, TCG target maintainers would be the main stakeholders. To > some extent, various Machine maintainers would also be stakeholders, > but they will most likely come back to TCG target maintainers looking > for solution. In a literal sense, a number of maintainers were > initially going to be very unhappy seeing the results (for example, > seeing that the machine or entire target performs poorly compared to > similar machines/targets), but after a while they should and will > become happy realizing the problem was identified, and the culprit is > at least approximately determined. > > I intentionally wanted to keep the project description simple in order > to be realistic and not develop high expectation among any of us. And > if the student proved to be capable, it will be very easy to add some > more useful tasks for him in this area, to be included in his/hers > GSoC/Outreachy activities. > > He had just today one case of performance degradation identified manually: > > https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg06326.html > > This project aims to do these kind of things easier, and possibly in > an automated way. Howard did this by manual measurements for one > particular setup, but this project will cover much much more. > > Thanks, Stefan, again for your interest - and everything else!
Please go ahead and add this project idea to the wiki: https://wiki.qemu.org/Google_Summer_of_Code_2020#How_to_add_a_project_idea Stefan
signature.asc
Description: PGP signature
