Hi all, I have been looking into the bookkeeper build time for a while. It usually takes about 1hr ~ 2hrs to completely run a bookkeeper build with full tests.
Most of the time is spent on running bookkeeper-server module. Building bookkeeper-server module usually takes about 50 mins to 1 hr. Sometimes it also causes time out. E.g. https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java8/1178/org.apache.bookkeeper$bookkeeper-server/ ``` *Started 3 hr 29 min ago* *Took 57 min <https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java8/org.apache.bookkeeper$bookkeeper-server/buildTimeTrend> on H23 <https://builds.apache.org/computer/H23>* ``` Looking into the detailed build time, following 4 packages used most of the time. https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java8/1178/org.apache.bookkeeper$bookkeeper-server/testReport/ - `org.apache.bookkeeper.client` : 19min - `org.apache.bookkeeper.bookie` : 7min - `org.apache.bookkeeper.replication`: 7min - `org.apache.bookkeeper.tls`: 7min - all the remaining tests: 13min In order to improve the building time, we can split the build job into multiple smaller jobs that only run a subset of tests. so we can reduce the build time to 1/3 or even 1/4. Here is one quick prototype using travis: https://travis-ci.org/sijie/bookkeeper/builds/388083586 using `mvn -pl bookkeeper-server test -Dtest=${BK_TEST_GROUP}` to test a group of tests. Those test groups can be: - client: "org.apache.bookkeeper.client.**" - bookie: "org.apache.bookkeeper.bookie.**" - replication: "org.apache.bookkeeper.replication.**" - tls: "org.apache.bookkeeper.tls.**" - all others: "!org.apache.bookkeeper.client.**,!org.apache.bookkeeper.bookie.**,!org.apache.bookkeeper.replication.**,! org.apache.bookkeeper.tls.**" Thoughts? If this looks good, I will submit a PR for this, but do it in apache jenkins. - Sijie
