Thanks Yu for bringing up this discussion.
The e2e perf tests can be really helpful and the overall design looks good
to me.
Sorry it's late but I have 2 questions about the result check.
1. How do we measure the job throughput? By measuring the job execution
time on a finite input data set, or measuring the QPS when the job has
reached a stable state?
I ask this because that, with LazyFromSource schedule mode, tasks are
launched gradually on processing progress.
So if we are measuring the throughput in the latter way,
the LazyFromSource scheduling would make no difference with Eager
scheduling. So we can drop this dimension if taking this way.
By measuring the total execution time, however, it can be kept since the
scheduling effectiveness can make differences, especially in small input
data set cases.
2. In our prior experiences, the performance result is usually not that
stable, which may make the perf degradation harder to detect.
Shall we define the rounds to run a job and how to aggregate the
result, so that we can get a more reliable final performance result?
Thanks,
Zhu Zhu
Yu Li <car...@gmail.com> 于2019年11月14日周四 上午10:52写道：
> Since one week passed and no more comments, I assume the latest FLIP doc
> looks good to all and will open a VOTE thread soon for the FLIP. Thanks for
> all the comments and discussion!
>
> Best Regards,
> Yu
>
>
> On Thu, 7 Nov 2019 at 18:35, Yu Li <car...@gmail.com> wrote:
>
> > Thanks for the comments Biao!
> >
> > bq. It seems this proposal is separated into several stages. Is there a
> > more detailed plan?
> > Good point! For stage one we'd like to try introducing the benchmark
> > first, so we could guard the release (hopefully starting from 1.10). For
> > other stages, we don't have detailed plan yet, but will add child FLIPs
> > when moving on and open new discussion/voting separately. I have updated
> > the FLIP document to better reflect this, please check it and let me know
> > what you think. Thanks.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 5 Nov 2019 at 10:16, Biao Liu <mmyy1...@gmail.com> wrote:
> >
> >> Thanks Yu for bringing this topic.
> >>
> >> +1 for this proposal. Glad to have an e2e performance testing.
> >>
> >> It seems this proposal is separated into several stages. Is there a more
> >> detailed plan?
> >>
> >> Thanks,
> >> Biao /'bɪ.aʊ/
> >>
> >>
> >>
> >> On Mon, 4 Nov 2019 at 19:54, Congxian Qiu <qcx978132...@gmail.com>
> wrote:
> >>
> >> > +1 for this idea.
> >> >
> >> > Currently, we have the micro benchmark for flink, which can help us
> find
> >> > the regressions. And I think the e2e jobs performance testing can also
> >> help
> >> > us to cover more scenarios.
> >> >
> >> > Best,
> >> > Congxian
> >> >
> >> >
> >> > Jingsong Li <jingsongl...@gmail.com> 于2019年11月4日周一 下午5:37写道：
> >> >
> >> > > +1 for the idea. Thanks Yu for driving this.
> >> > > Just curious about that can we collect the metrics about Job
> >> scheduling
> >> > and
> >> > > task launch. the speed of this part is also important.
> >> > > We can add tests for watch it too.
> >> > >
> >> > > Look forward to more batch test support.
> >> > >
> >> > > Best,
> >> > > Jingsong Lee
> >> > >
> >> > > On Mon, Nov 4, 2019 at 10:00 AM OpenInx <open...@gmail.com> wrote:
> >> > >
> >> > > > > The test cases are written in java and scripts in python. We
> >> propose
> >> > a
> >> > > > separate directory/module in parallel with flink-end-to-end-tests,
> >> with
> >> > > the
> >> > > > > name of flink-end-to-end-perf-tests.
> >> > > >
> >> > > > Glad to see that the newly introduced e2e test will be written in
> >> Java.
> >> > > > because I'm re-working on the existed e2e tests suites from BASH
> >> > scripts
> >> > > > to Java test cases so that we can support more external system ,
> >> such
> >> > as
> >> > > > running the testing job on yarn+flink, docker+flink,
> >> standalone+flink,
> >> > > > distributed kafka cluster etc.
> >> > > > BTW, I think the perf e2e test suites will also need to be
> designed
> >> as
> >> > > > supporting running on both standalone env and distributed env.
> will
> >> be
> >> > > > helpful
> >> > > > for developing & evaluating the perf.
> >> > > > Thanks.
> >> > > >
> >> > > > On Mon, Nov 4, 2019 at 9:31 AM aihua li <liaihua1...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > In stage1, the checkpoint mode isn't disabled,and uses heap as
> the
> >> > > > > statebackend.
> >> > > > > I think there should be some special scenarios to test
> checkpoint
> >> and
> >> > > > > statebackend, which will be discussed and added in the
> >> release-1.11
> >> > > > >
> >> > > > > > 在 2019年11月2日，上午12:13，Yun Tang <myas...@live.com> 写道：
> >> > > > > >
> >> > > > > > By the way, do you think it's worthy to add a checkpoint mode
> >> which
> >> > > > just
> >> > > > > disable checkpoint to run end-to-end jobs? And when will stage2
> >> and
> >> > > > stage3
> >> > > > > be discussed in more details?
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Best, Jingsong Lee
> >> > >
> >> >
> >>
> >
>
