Hi Yanfei, Thanks for driving the benchmark monitoring effort! The Google doc and the community wiki looks pretty good.
According to Yuan's comment, it seems that we currently manually watch the benchmark results to detect regression. Have we considered automating this process by e.g. exporting the nightly benchmark results to a database and using scripts to detect regression based on pre-defined rules? This approach is probably more scalable and accurate in the long term. And I had a good experience working with such a regression detection tool in my past job. Thanks, Dong On Thu, Jan 19, 2023 at 4:02 PM Yanfei Lei <fredia...@gmail.com> wrote: > Hi devs, > > I'd like to start a discussion about incorporating performance > regression monitoring into the routine process. Flink benchmarks are > periodically executed on http://codespeed.dak8s.net:8080 to monitor > Flink performance. In late Oct'22, a new slack channel > #flink-dev-benchmarks was created for notifications of performance > regressions. It helped us find 2 build failures[1,2] and 5 performance > regressions[3,4,5,6,7] in the past 3 months, which is very meaningful > to ensuring the quality of the code. > > There are some release managers( cc @Matthias, @Martijn, @Qingsheng) > proposing to incorporate performance regression monitoring into the > release management, I think it makes sense for performance stabilities > (like CI stabilities), since almost every release has some tickets > about performance optimizations, the performance monitoring can > effectively avoid performance regression and track the performance > improvement of each release. So I start this discussion to pick > everyone’s brain for some suggestions. > > In the past, I checked the slack notifications once a week, and I have > summarized a draft[8]( > https://docs.google.com/document/d/1jTTJHoCTf8_LAjviyAY3Fi7p-tYtl_zw7rJKV4V6T_c/edit?usp=sharing > ) > on how to deal with performance regressions according to some > contributors and my own experience. If the above proposal is > considered acceptable, I’d like to put it in the community wiki[9]. > > Looking forward to your feedback! > > [1] https://issues.apache.org/jira/browse/FLINK-29883 > [2] https://issues.apache.org/jira/browse/FLINK-30015 > [3] https://issues.apache.org/jira/browse/FLINK-29886 > [4] https://issues.apache.org/jira/browse/FLINK-30181 > [5] https://issues.apache.org/jira/browse/FLINK-30623 > [6] https://issues.apache.org/jira/browse/FLINK-30624 > [7] https://issues.apache.org/jira/browse/FLINK-30625 > [8] > https://docs.google.com/document/d/1jTTJHoCTf8_LAjviyAY3Fi7p-tYtl_zw7rJKV4V6T_c/edit?usp=sharing > [9] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847 > > Best, > Yanfei >
