Hi, Per updates on VPP and CSIT project calls, few of us in CSIT worked on getting the VPP performance trending into a usable shape. We got there in the end, initial (experimental!) version is out. Thanks to Peter, Tibor and Vratko for great work, I was just complaining :)
The results are published as follows: 1. VPP Performance Dashboard [1] - provides the latest VPP throughput trend, trend compliance and detected anomalies, all on a per VPP test case basis. Test cases are clickable, leading to associated trendline graph. Regressions are listed first (red), followed by progressions (green), outliers (grey) and normal. 2. Trending Methodology [2] - describes used performance test metrics and calculations for performance trend and anomaly classification (progress, regression, outlier). 3. Trendline Graphs [3] - show graphically sequence of measurements against the trendline, and anomaly classification, with associated CSIT test jobs and VPP builds. Note this is still in experimental phase, so don’t be alarmed if you see your VPP code packet path being listed in red - it could always be a false positive :) Same for code paths listed in green.. All feedback welcome, especially re anomaly detection accuracy, usability and clarity of presentation and drill-down. We know there is a lot that could be done here to improve, so constructive feedback is golden :) Below also a list of few caveats. -Maciek [1] https://docs.fd.io/csit/master/trending/introduction/index.html [2] https://docs.fd.io/csit/master/trending/methodology/index.html [3] https://docs.fd.io/csit/master/trending/trending/index.html Caveats: 1. Update frequency - all dynamic content is to be updated every ~12hrs, but right now it’s every other day due to LF testbeds occupied with rls1804 testing work, back to daily next week. 2. Following test cases are currently not present in trendline graphs: (listing only 1t1c variants, 2t2c, 4t4c variants are also affected) 10ge2p1x520-imix-1t1c-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr 10ge2p1x520-64b-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr 10ge2p1x520-imix-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr 10ge2p1x520-64b-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr 10ge2p1x520-imix-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr 10ge2p1x710-64b-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr 10ge2p1x710-imix-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr 10ge2p1x710-64b-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr 10ge2p1x710-imix-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr 10ge2p1xl710-64b-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr 10ge2p1xl710-imix-1t1c-eth-l2xcbase-eth-4vhostvr1024-2vm-ndrdisc-mrr 10ge2p1xl710-64b-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr 10ge2p1xl710-imix-1t1c-eth-l2bdbasemaclrn-eth-4vhostvr1024-2vm-mrr 3. Following test cases shouldn’t be listed in dashboard tables: 10ge2p1x520-64b-1t1c-eth-l2bdscale1mmaclrn-ndrdisc 10ge2p1x520-64b-2t2c-eth-l2bdscale1mmaclrn-ndrdisc 10ge2p1x520-64b-4t4c-eth-l2bdscale1mmaclrn-ndrdisc
