Agreed with your remarks regarding the unreliability of benchmark results in the cloud. See my proposal in private@ to get some machines for continuous benchmarks.
On Tue, Dec 28, 2021 at 10:17 AM Dominik Psenner <[email protected]> wrote: > Hi Stephen, > > The trouble with benchmarks in CI is that the numbers may be largely > unreliable, depending mostly on the hardware where it runs and in general > the surrounding environment. Chances are high that the benchmarks will not > produce comparable results. > > It would however be good to provide some tools to run the (same) benchmarks > manually. > > When run on the same hardware with different codebases or on different > hardware with the same codebase, the outcome may provide interesting and > comparable insights. > > Warm regards > -- > Sent from my phone. Typos are a kind gift to anyone who happens to find > them. > > On Tue, Dec 28, 2021, 07:46 Stephen Webb <[email protected]> wrote: > > > Hi, > > > > Robert has created a benchmark that I thought would be nice to integrate > > into CI. > > > > I see the Log4J has some benchmarks actions which are currently run > > manually with results posted to github pages. > > > > Do you consider this a useful/optimal approach? > > > > Would an threshold which an action could check for each PR be useful? > > > > Regards > > Stephen Webb > > > > < > > > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > Virus-free. > > www.avast.com > > < > > > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > >
