+1 for java-based E2E test as bash scripts lack of the power to handle more complicated cases.
For the docker image improvement, I think we should be more cautious as developers at China might suffer with the network issue, or at least we should ensure or find some guides to speed up the image downloading. Best Yun Tang ________________________________ From: Xingbo Huang <hxbks...@gmail.com> Sent: Thursday, November 19, 2020 12:09 To: dev <dev@flink.apache.org> Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink Big +1 for java-based e2e tests. Currently PyFlink related tests each take ~15minutes in bash e2e tests because we are using a secured YARN cluster which is the only convenient way of starting a YARN cluster in the bash e2e tests. I think if we migrate these tests to the java-based testing framework, we will start a Yarn Cluster more conveniently, which will greatly reduce our testing time. Best, Xingbo Rui Li <lirui.fu...@gmail.com> 于2020年11月19日周四 上午10:47写道： > Big +1 to java-based e2e tests. It'll be much easier to write/debug these > tests. > > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xbjt...@gmail.com> wrote: > > > +1 to stop using bash scripts, > > and I also have experienced the bash scripts that is really hard to > > maintain and debug, thanks @Robert for the great work again. > > > > I think testcontainers is a nice candidate. > > > > Best, > > Leonard > > > > > 在 2020年11月18日，19:46，Aljoscha Krettek <aljos...@apache.org> 写道： > > > > > > +1 > > > > > > And I want to second Arvid's mention of testcontainers [1]. > > > > > > [1] https://www.testcontainers.org/ > > > > > > On 18.11.20 10:43, Yang Wang wrote: > > >> Thanks till and Jark for sharing the information. > > >> I am also +1 for this proposal and glad to wire the new introduced K8s > > HA > > >> e2e tests to java based framework. > > >> Best, > > >> Yang > > >> Jark Wu <imj...@gmail.com> 于2020年11月18日周三 下午5:23写道： > > >>> +1 to use the Java-based testing framework and +1 for using docker > > images > > >>> in the future. > > >>> IIUC, the Java-based testing framework refers to the > > >>> `flink-end-to-end-tests-common` module. > > >>> The java-based framework helped us a lot when debugging the unstable > > e2e > > >>> tests. > > >>> > > >>> Best, > > >>> Jark > > >>> > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <danrtsey...@gmail.com> > wrote: > > >>> > > >>>> Thanks for starting this discussion. > > >>>> > > >>>> In general, I agree with you that a java-based testing framework is > > >>> better > > >>>> than the bash-based. It will > > >>>> help a lot for the commons and utilities. > > >>>> > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have > > some > > >>>> quick questions. > > >>>> * I am not sure where the java-based framework is. Do you mean > > >>>> "flink-jepsen" module or sth else? > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run / > > >>>> run-application) to submit a Flink job in the java-based framework. > > >>>> * It will be harder to inject some operations. For example, kill the > > >>>> JobManager in Kubernetes. Currently, I > > >>>> am trying to use "kubectl exec" to do this. > > >>>> > > >>>> > > >>>> Best, > > >>>> Yang > > >>>> > > >>>> Robert Metzger <rmetz...@apache.org> 于2020年11月17日周二 下午11:36写道： > > >>>> > > >>>>> Hi all, > > >>>>> > > >>>>> Since we are currently testing the 1.12 release, and potentially > > adding > > >>>>> more automated e2e tests, I would like to bring up our end to end > > tests > > >>>> for > > >>>>> discussion. > > >>>>> > > >>>>> Some time ago, we introduced a Java-based testing framework, with > the > > >>>> idea > > >>>>> of replacing the current bash-based end to end tests. > > >>>>> Since the introduction of the java-based framework, more bash tests > > >>> were > > >>>>> actually added, making a future migration even harder. > > >>>>> > > >>>>> *For that reason, I would like to propose that we are stopping to > add > > >>> any > > >>>>> new bash end to end tests to Flink. All new end to end tests must > be > > >>>>> written in Java and rely on the existing testing framework.* > > >>>>> > > >>>>> For the 1.13 release, I'm trying to find some time to revisit > > potential > > >>>>> improvements for the existing java e2e framework (such as using > > Docker > > >>>>> images everywhere), as well as a migration plan for the existing > bash > > >>>>> tests. We have a large number of bash e2e tests that are just > > >>>> parameterized > > >>>>> differently. If we would start migrating them to Java, we could > move > > a > > >>>>> larger proportion of tests over to the new Java framework, and > tackle > > >>> the > > >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...). > > >>>>> > > >>>>> Let me know what you think! > > >>>>> > > >>>>> Best, > > >>>>> Robert > > >>>>> > > >>>>> > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending > > >>>> quite a > > >>>>> lot of time trying to figure out really hard to debug issues with > our > > >>>> bash > > >>>>> testing infra. > > >>>>> Also, it is very difficult to introduce something generic for all > > tests > > >>>>> (such as a test-timeout, using docker as the preferred deployment > > >>> method > > >>>>> etc.) since the tests often don't share common tooling. > > >>>>> Speaking about tooling: there are a lot of utilities everywhere, > > >>>> sometimes > > >>>>> duplicated, with different features / stability etc. > > >>>>> I believe bash is not the right tool for a project this size (in > > terms > > >>> of > > >>>>> developers and lines of code) > > >>>>> > > >>>> > > >>> > > > > > > > > > -- > Best regards! > Rui Li >
