Hello VPP devs. Now that CSIT testbeds are not as busy running tests for 1810 release report, you can give the per-patch performance job a try again.
The job can still be used as announced (see below), but the internal logic has been tweaked. Good news: False regressions should be much less frequent now. Bad news: One run takes 90-100 minutes (not counting queueing time). The job still gets the following comment on failure: No problems were identified. If you know why this problem occurred, please add a suitable Cause for it. How can I add a Cause to distinguish "Regression has been detected" from other (rare but still possible) failures? By the way, while testing with [0] I have confirmed that the previously merged [1] causes (some of) the regressions seen in trending. Unfortunately, the changes merged in the meantime make it hard to revert it. Vratko. [0] https://gerrit.fd.io/r/15826 [1] https://gerrit.fd.io/r/15347 From: csit-...@lists.fd.io <csit-...@lists.fd.io> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via Lists.Fd.Io Sent: Friday, 2018-September-07 17:20 To: vpp-dev@lists.fd.io Cc: csit-...@lists.fd.io Subject: Re: [csit-dev] new per-patch job > and vote -1 if a regression is detected. Original plan was to have the job non-voting. First implementation has left the voting active by accident, now it is switched to non-voting (also by accident). Non-voting is safer, we can be sure result of this job does not override possible -1 from other verify jobs. Vratko. From: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) Sent: Friday, 2018-September-07 15:25 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Cc: csit-dev <csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>> Subject: RE: new per-patch job I forgot to mention two things. > When you add Gerrit comment "perftest" We can make the job to run on every uploaded patch (if not Draft) when it is deemed reliable enough. > three main areas for improvements: Longer term goal is to add some logic to select testcases based on directories edited. The hard part is creating a mapping from directories to test tags. Vratko. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via Lists.Fd.Io Sent: Friday, 2018-September-07 15:04 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] new per-patch job Hello VPP devs. There is a new way of testing performance impact of a VPP Change before merging it. When you add Gerrit comment "perftest" a job will start (different from the previously available one). It will compile both the code in the Change and in its parent, run few tests against both builds, and vote -1 if a regression is detected. The job takes around one hour to finish (similar to virl job), mostly spent by compiling. The job compares results of 4 tests by default: ip4base, ip6base, l2bdbasemaclrn and l2xcbase. The job supports tag selection (the same way as the previous job). The job is using 2-node skylake testbeds, so some tests are not available (e.g. the ones related to encapsulation). The regressions are detected using the same algorithm as in trending graphs, but the result is not 100% reliable. Sometimes, even running the same build twice shows statistically significant difference in performance. Going forward, there are three main areas for improvements: 1. Add more tests (to the set executed by default). This would find more kinds of performance regressions. 2. Make the job run faster. Some kind of artifact caching could be implemented, possibly using packagecloud. 3. Make the vote more reliable. Either by identifying and fixing the current causes of performance differences, or at least by working around them (somehow). So please, give the new job a try, and tell us which area of improvement to focus on. Vratko (and CSIT team).
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11203): https://lists.fd.io/g/vpp-dev/message/11203 Mute This Topic: https://lists.fd.io/mt/28090222/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
