Thanks Yu for starting this discussion. I'm in favor of adding a e2e performance testing framework. Currently the e2e tests are mainly focused on functionality and written in shell. We need a better e2e framework for performance and functionality tests.
Best, Yang Biao Liu <mmyy1...@gmail.com> 于2019年11月5日周二 上午10:16写道： > Thanks Yu for bringing this topic. > > +1 for this proposal. Glad to have an e2e performance testing. > > It seems this proposal is separated into several stages. Is there a more > detailed plan? > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Mon, 4 Nov 2019 at 19:54, Congxian Qiu <qcx978132...@gmail.com> wrote: > > > +1 for this idea. > > > > Currently, we have the micro benchmark for flink, which can help us find > > the regressions. And I think the e2e jobs performance testing can also > help > > us to cover more scenarios. > > > > Best, > > Congxian > > > > > > Jingsong Li <jingsongl...@gmail.com> 于2019年11月4日周一 下午5:37写道： > > > > > +1 for the idea. Thanks Yu for driving this. > > > Just curious about that can we collect the metrics about Job scheduling > > and > > > task launch. the speed of this part is also important. > > > We can add tests for watch it too. > > > > > > Look forward to more batch test support. > > > > > > Best, > > > Jingsong Lee > > > > > > On Mon, Nov 4, 2019 at 10:00 AM OpenInx <open...@gmail.com> wrote: > > > > > > > > The test cases are written in java and scripts in python. We > propose > > a > > > > separate directory/module in parallel with flink-end-to-end-tests, > with > > > the > > > > > name of flink-end-to-end-perf-tests. > > > > > > > > Glad to see that the newly introduced e2e test will be written in > Java. > > > > because I'm re-working on the existed e2e tests suites from BASH > > scripts > > > > to Java test cases so that we can support more external system , such > > as > > > > running the testing job on yarn+flink, docker+flink, > standalone+flink, > > > > distributed kafka cluster etc. > > > > BTW, I think the perf e2e test suites will also need to be designed > as > > > > supporting running on both standalone env and distributed env. will > be > > > > helpful > > > > for developing & evaluating the perf. > > > > Thanks. > > > > > > > > On Mon, Nov 4, 2019 at 9:31 AM aihua li <liaihua1...@gmail.com> > wrote: > > > > > > > > > In stage1, the checkpoint mode isn't disabled,and uses heap as the > > > > > statebackend. > > > > > I think there should be some special scenarios to test checkpoint > and > > > > > statebackend, which will be discussed and added in the release-1.11 > > > > > > > > > > > 在 2019年11月2日，上午12:13，Yun Tang <myas...@live.com> 写道： > > > > > > > > > > > > By the way, do you think it's worthy to add a checkpoint mode > which > > > > just > > > > > disable checkpoint to run end-to-end jobs? And when will stage2 and > > > > stage3 > > > > > be discussed in more details? > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best, Jingsong Lee > > > > > >
