Hi, This seems like a good task for us to do. I will see what it would take in order to convert the difference into a decimal-formatted percentage. I have put this into bugzilla to keep track of this issue: https://bugs.dpdk.org/show_bug.cgi?id=626
Thanks, Brandon On Sat, Jan 23, 2021 at 3:57 AM Morten Brørup <m...@smartsharesystems.com> wrote: > > > From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Lincoln Lavoie > > Sent: Thursday, January 21, 2021 5:35 PM > > To: Morten Brørup > > > > Hi All, > > > > Trying to follow the specific conversation. It is correct, the lab > > does > > not list the specific throughput values achieved by the hardware, as > > that > > data can be sensitive to the hardware vendors, etc. The purpose of the > > lab > > is to check for degradations caused by patches, so the difference is > > really > > the important factor. The comparison is against a prior run on the > > same > > hardware, via the DPDK main branch, so any delta should be caused by > > the > > specific patch changes (excluding statistical "wiggle"). > > > > Thank you for clarifying, Lincoln. > > This sounds like a perfect solution to the meet the purpose. > > I request that you change the output to show the relative difference (i.e. > percentage above/below baseline), instead of the absolute difference (i.e. > number of packets per second). > > By showing a percentage, anyone reading the test report can understand if the > difference is insignificant, or big enough to require further discussion > before accepting the patch. Showing the difference in packets per second > requires the reader of the test report to have prior knowledge about the > expected performance. > > > If the group would prefer, we could calculate additional references if > > desired (i.e. difference from the last official release, or a monthly > > run > > of the current, etc.). We just need the community to define their > > needs, > > and we can add this to the development queue. > > > > I retract my suggestion about adding additional references. You clearly > explained how your baselining works, and I think it fully serves the needs of > a regression test. > > So please just put this small change in your development queue: Output the > difference in percent with a few decimals after the comma, instead of the > difference in packets per second. > > For readability, performance drops should be shown as negative values, and > increases as positive, e.g.: > > Difference = (NewPPS - BaselinePPS) / BaselinePPS * 100.00 %. > > > If we want to compare performance against official releases, current or > older, it should go elsewhere, not in the continuous testing environment. > E.g. the release notes could include a report showing differences from the > last few official releases. But that is a task for another day. > > > > Cheers, > > Lincoln > > > > > > On Thu, Jan 21, 2021 at 4:29 AM Morten Brørup > > <m...@smartsharesystems.com> > > wrote: > > > > > > From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Ferruh Yigit > > > > Sent: Thursday, January 21, 2021 10:19 AM > > > > > > > > On 1/15/2021 6:39 PM, Ali Alnubani wrote: > > > > > Hi, > > > > > Adding Ferruh and Zhaoyan, > > > > > > > > > >> Ali, > > > > >> > > > > >> You reported some performance regression, did you confirm it? > > > > >> If I get no reply by monday, I'll proceed with this patch. > > > > > > > > > > Sure I'll confirm by Monday. > > > > > > > > > > Doesn't the regression also reproduce on the Lab's Intel servers? > > > > > Even though the check iol-intel-Performance isn't failing, I can > > see > > > > that the throughput differences from expected for this patch are > > less > > > > than those of another patch that was tested only 20 minutes > > earlier. > > > > Both patches were applied to the same tree: > > > > > > > > > > https://mails.dpdk.org/archives/test-report/2021- > > January/173927.html > > > > >> | 64 | 512 | 1.571 | > > > > > > > > > > https://mails.dpdk.org/archives/test-report/2021- > > January/173919.html > > > > >> | 64 | 512 | 2.698 | > > > > > > > > > > Assuming that pw86457 doesn't have an effect on this test, it > > looks > > > > to me that this patch caused a regression in Intel hardware as > > well. > > > > > > > > > > Can someone update the baseline's expected values for the Intel > > NICs > > > > and rerun the test on this patch? > > > > > > > > > > > > > Zhaoyan said that the baseline is calculated dynamically, > > > > what I understand is baseline set based on previous days > > performance > > > > result, so > > > > it shouldn't require updating. > > > > > > That sounds smart! > > > > > > Perhaps another reference baseline could be added, for informational > > > purposes only: > > > Deviation from the performance of the last official release. > > > > > > > > > > > But cc'ed the lab for more details. > > > > > > > > > > -- > > *Lincoln Lavoie* > > Senior Engineer, Broadband Technologies > > 21 Madbury Rd., Ste. 100, Durham, NH 03824 > > lylav...@iol.unh.edu > > https://www.iol.unh.edu > > +1-603-674-2755 (m) > > <https://www.iol.unh.edu> > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 b...@iol.unh.edu www.iol.unh.edu
