Thanks Robert for pushing this. +1 for creating java-based e2e tests. In the engine team, we decided to work towards the goal of migrating the bash-based e2e tests to Java/Docker. We plan to migrate the existing bash-based e2e tests located in the Engine team's component space step-by-step. I created an umbrella Jira issue [1] to collect and document the migration efforts. Feel free to do the same by creating subtasks under [1].
Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-20392 On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <myas...@live.com> wrote: > +1 for java-based E2E test as bash scripts lack of the power to handle > more complicated cases. > > For the docker image improvement, I think we should be more cautious as > developers at China might suffer with the network issue, or at least we > should ensure or find some guides to speed up the image downloading. > > Best > Yun Tang > ________________________________ > From: Xingbo Huang <hxbks...@gmail.com> > Sent: Thursday, November 19, 2020 12:09 > To: dev <dev@flink.apache.org> > Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink > > Big +1 for java-based e2e tests. Currently PyFlink related tests each take > ~15minutes in bash e2e tests because we are using a secured YARN cluster > which is the only convenient way of starting a YARN cluster in the bash e2e > tests. I think if we migrate these tests to the java-based testing > framework, we will start a Yarn Cluster more conveniently, which will > greatly reduce our testing time. > > Best, > Xingbo > > Rui Li <lirui.fu...@gmail.com> 于2020年11月19日周四 上午10:47写道： > > > Big +1 to java-based e2e tests. It'll be much easier to write/debug these > > tests. > > > > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xbjt...@gmail.com> wrote: > > > > > +1 to stop using bash scripts, > > > and I also have experienced the bash scripts that is really hard to > > > maintain and debug, thanks @Robert for the great work again. > > > > > > I think testcontainers is a nice candidate. > > > > > > Best, > > > Leonard > > > > > > > 在 2020年11月18日，19:46，Aljoscha Krettek <aljos...@apache.org> 写道： > > > > > > > > +1 > > > > > > > > And I want to second Arvid's mention of testcontainers [1]. > > > > > > > > [1] https://www.testcontainers.org/ > > > > > > > > On 18.11.20 10:43, Yang Wang wrote: > > > >> Thanks till and Jark for sharing the information. > > > >> I am also +1 for this proposal and glad to wire the new introduced > K8s > > > HA > > > >> e2e tests to java based framework. > > > >> Best, > > > >> Yang > > > >> Jark Wu <imj...@gmail.com> 于2020年11月18日周三 下午5:23写道： > > > >>> +1 to use the Java-based testing framework and +1 for using docker > > > images > > > >>> in the future. > > > >>> IIUC, the Java-based testing framework refers to the > > > >>> `flink-end-to-end-tests-common` module. > > > >>> The java-based framework helped us a lot when debugging the > unstable > > > e2e > > > >>> tests. > > > >>> > > > >>> Best, > > > >>> Jark > > > >>> > > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <danrtsey...@gmail.com> > > wrote: > > > >>> > > > >>>> Thanks for starting this discussion. > > > >>>> > > > >>>> In general, I agree with you that a java-based testing framework > is > > > >>> better > > > >>>> than the bash-based. It will > > > >>>> help a lot for the commons and utilities. > > > >>>> > > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I > have > > > some > > > >>>> quick questions. > > > >>>> * I am not sure where the java-based framework is. Do you mean > > > >>>> "flink-jepsen" module or sth else? > > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run / > > > >>>> run-application) to submit a Flink job in the java-based > framework. > > > >>>> * It will be harder to inject some operations. For example, kill > the > > > >>>> JobManager in Kubernetes. Currently, I > > > >>>> am trying to use "kubectl exec" to do this. > > > >>>> > > > >>>> > > > >>>> Best, > > > >>>> Yang > > > >>>> > > > >>>> Robert Metzger <rmetz...@apache.org> 于2020年11月17日周二 下午11:36写道： > > > >>>> > > > >>>>> Hi all, > > > >>>>> > > > >>>>> Since we are currently testing the 1.12 release, and potentially > > > adding > > > >>>>> more automated e2e tests, I would like to bring up our end to end > > > tests > > > >>>> for > > > >>>>> discussion. > > > >>>>> > > > >>>>> Some time ago, we introduced a Java-based testing framework, with > > the > > > >>>> idea > > > >>>>> of replacing the current bash-based end to end tests. > > > >>>>> Since the introduction of the java-based framework, more bash > tests > > > >>> were > > > >>>>> actually added, making a future migration even harder. > > > >>>>> > > > >>>>> *For that reason, I would like to propose that we are stopping to > > add > > > >>> any > > > >>>>> new bash end to end tests to Flink. All new end to end tests must > > be > > > >>>>> written in Java and rely on the existing testing framework.* > > > >>>>> > > > >>>>> For the 1.13 release, I'm trying to find some time to revisit > > > potential > > > >>>>> improvements for the existing java e2e framework (such as using > > > Docker > > > >>>>> images everywhere), as well as a migration plan for the existing > > bash > > > >>>>> tests. We have a large number of bash e2e tests that are just > > > >>>> parameterized > > > >>>>> differently. If we would start migrating them to Java, we could > > move > > > a > > > >>>>> larger proportion of tests over to the new Java framework, and > > tackle > > > >>> the > > > >>>>> more involved bash tests later (kerberized yarn, kubernetes, > ...). > > > >>>>> > > > >>>>> Let me know what you think! > > > >>>>> > > > >>>>> Best, > > > >>>>> Robert > > > >>>>> > > > >>>>> > > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm > spending > > > >>>> quite a > > > >>>>> lot of time trying to figure out really hard to debug issues with > > our > > > >>>> bash > > > >>>>> testing infra. > > > >>>>> Also, it is very difficult to introduce something generic for all > > > tests > > > >>>>> (such as a test-timeout, using docker as the preferred deployment > > > >>> method > > > >>>>> etc.) since the tests often don't share common tooling. > > > >>>>> Speaking about tooling: there are a lot of utilities everywhere, > > > >>>> sometimes > > > >>>>> duplicated, with different features / stability etc. > > > >>>>> I believe bash is not the right tool for a project this size (in > > > terms > > > >>> of > > > >>>>> developers and lines of code) > > > >>>>> > > > >>>> > > > >>> > > > > > > > > > > > > > > -- > > Best regards! > > Rui Li > > > -- Matthias Pohl | Engineer Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton Wehner
