On Mon, Feb 11, 2019 at 12:12 AM Matteo Merli <mme...@apache.org> wrote:
> Agree the split will be a good idea. > > Anyway I think the "bigger" problem is currently with the Java unit > tests job. This job takes ~90min and has currently the higher chances > of failure. Most of time (and all flakyness) is in the broker tests > module which is essentially used as a mid-way between unit and > integration tests. > I'm trying to come up with some reasonable plan to gradually improve > the situation there, without losing sanity... Agreed and +1 on improving java8 test job > > Regarding integration, I think that most of time is being spent in > starting the containers for each new test class. > Since the integration tests are already in a (very) much better state, > compared to our unit tests, I think that perhaps we could have a > single cluster, statically initialized that can be shared across many > tests, each using its own namespaces/topics. Current integration is already doing the approach as you described. For each test suite, it initializes the a cluster before running all tests, and reuses the cluster for all tests during the whole suite. For each test, we use a separate namespace. I am not sure what will be the difference in your proposal. My proposal is more about breaking down the integration test into multiple smaller jobs. So when a failure occurs, it doesn’t have to rerun the whole test. We have used this breakdown approach in bookkeeper’s CI, which has been running very reliably. > > This way we can still have multiple test classes (which makes > understanding the code and the results easier) and only pay the > startup price once. > > Finally, this will be easier to start with only tests that don't > involve killing broker containers, so probably the Jenkins jobs could > be split based on that. > > > On Sat, Feb 9, 2019 at 10:47 AM Eren Avsarogullari > <erenavsarogull...@gmail.com> wrote: > > > > +1 > > > > Also, i have suggested 'quarantined-list' feature for Apache Heron to > > manage flaky integration-tests as follow. As the long-term, it can also > be > > useful for Pulsar if similar feature is already not used. > > https://github.com/apache/incubator-heron/issues/2865 > > > > On Sat, 9 Feb 2019 at 04:54, Sanjeev Kulkarni <sanjee...@gmail.com> > wrote: > > > > > +1 > > > > > > On Fri, Feb 8, 2019 at 8:52 PM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > > `Hi all, > > > > > > > > Integration job has been a pain point for merging pull requests. The > > > total > > > > run time of the integration job typically is around an hour. If an > > > > integration test is failing, retrigger the job requires another hour > to > > > > run. rerunning the job will take up the docker resources on jenkins > > > nodes. > > > > > > > > I am thinking of breaking down the current integration job into > multiple > > > > smaller jobs. This can achieve by specifying different test suite > files > > > > using system property. I have an outstanding pull request to > introduce > > > an ` > > > > integrationTestSuiteFile` system property. > > > > https://github.com/apache/pulsar/pull/3558/ > > > > > > > > The initial set of test suites I can think of are: > > > > > > > > - cli test suite: all cli related tests > > > > - function thread test suite: all tests related functions in thread > mode > > > > - function process test suite: all tests related functions in process > > > mode > > > > - sql test suite: pulsar sql related tests > > > > - storage test suite: tiered storage related tests. > > > > > > > > Any thoughts? If this is a good direction to go, I can break down the > > > > integration job once PR#3558 is merged. > > > > > > > > - Sijie > > > > > > > >
