Hi Yu, Thanks for bringing this up.
+1 for the idea and the proposal from my side. I think that the proposed Test Job List might be a bit redundant/excessive, but: - we can always adjust this later, once we have the infrastructure in place - as long as we have the computing resources and ability to quickly interpret the results/catch regressions, it doesn’t hurt to have more benchmarks/tests then strictly necessary. Which brings me to a question. How are you planning to execute the end-to-end benchmarks and integrate them with our build process? Another smaller question: > In this initial stage we will only monitor and display job throughput and > latency. Are you planning to monitor the throughput and latency at the same time? It might be a bit problematic, as when measuring the throughput you want to saturate the system and hit some bottleneck, which will cause a back-pressure (measuring latency at the same time when system is back pressured doesn’t make much sense). Piotrek > On 30 Oct 2019, at 11:54, Yu Li <car...@gmail.com> wrote: > > Hi everyone, > > We would like to propose FLIP-83 that adds an end-to-end performance > testing framework for Flink. We discovered some potential problems through > such an internal end-to-end performance testing framework before the > release of 1.9.0 [1], so we'd like to contribute it to Flink community as a > supplement to the existing daily run micro performance benchmark [2] and > nightly run end-to-end stability test [3]. > > The FLIP document could be found here: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework > > Please kindly review the FLIP document and let us know if you have any > comments/suggestions, thanks! > > [1] https://s.apache.org/m8kcq > [2] https://github.com/dataArtisans/flink-benchmarks > [3] https://github.com/apache/flink/tree/master/flink-end-to-end-tests > > Best Regards, > Yu
