Thanks Chesnay for bringing up this discussion and sharing those thoughts
to speed up the building process.
I'd +1 for option 2 and 3.
We can benefits a lot from Option 2. Developing table, connectors,
libraries, docs modules would result in much fewer tests(1/3 to 1/tens) to
run.
PRs for those modules take up more than half of all the PRs in my
observation.
Option 3 can be a supplementary to option 2 that if the PR is modifying
fundamental modules like flink-core or flink-runtime.
It can even be a switch of the tests scope(basic/full) of a PR, so that
committers do not need to trigger it multiple times.
With it we can postpone the testing of IT cases or connectors before the PR
reaches a stable state.
Thanks,
Zhu Zhu
Chesnay Schepler <ches...@apache.org> 于2019年8月15日周四 下午3:38写道：
> Hello everyone,
>
> improving our build times is a hot topic at the moment so let's discuss
> the different ways how they could be reduced.
>
>
> Current state:
>
> First up, let's look at some numbers:
>
> 1 full build currently consumes 5h of build time total ("total time"),
> and in the ideal case takes about 1h20m ("run time") to complete from
> start to finish. The run time may fluctuate of course depending on the
> current Travis load. This applies both to builds on the Apache and
> flink-ci Travis.
>
> At the time of writing, the current queue time for PR jobs (reminder:
> running on flink-ci) is about 30 minutes (which basically means that we
> are processing builds at the rate that they come in), however we are in
> an admittedly quiet period right now.
> 2 weeks ago the queue times on flink-ci peaked at around 5-6h as
> everyone was scrambling to get their changes merged in time for the
> feature freeze.
>
> (Note: Recently optimizations where added to ci-bot where pending builds
> are canceled if a new commit was pushed to the PR or the PR was closed,
> which should prove especially useful during the rush hours we see before
> feature-freezes.)
>
>
> Past approaches
>
> Over the years we have done rather few things to improve this situation
> (hence our current predicament).
>
> Beyond the sporadic speedup of some tests, the only notable reduction in
> total build times was the introduction of cron jobs, which consolidated
> the per-commit matrix from 4 configurations (different scala/hadoop
> versions) to 1.
>
> The separation into multiple build profiles was only a work-around for
> the 50m limit on Travis. Running tests in parallel has the obvious
> potential of reducing run time, but we're currently hitting a hard limit
> since a few modules (flink-tests, flink-runtime,
> flink-table-planner-blink) are so loaded with tests that they nearly
> consume an entire profile by themselves (and thus no further splitting
> is possible).
>
> The rework that introduced stages, at the time of introduction, did also
> not provide a speed up, although this changed slightly once more
> profiles were added and some optimizations to the caching have been made.
>
> Very recently we modified the surefire-plugin configuration for
> flink-table-planner-blink to reuse JVM forks for IT cases, providing a
> significant speedup (18 minutes!). So far we have not seen any negative
> consequences.
>
>
> Suggestions
>
> This is a list of /all /suggestions for reducing run/total times that I
> have seen recently (in other words, they aren't necessarily mine nor may
> I agree with all of them).
>
> 1. Enable JVM reuse for IT cases in more modules.
> * We've seen significant speedups in the blink planner, and this
> should be applicable for all modules. However, I presume there's
> a reason why we disabled JVM reuse (information on this would be
> appreciated)
> 2. Custom differential build scripts
> * Setup custom scripts for determining which modules might be
> affected by change, and manipulate the splits accordingly. This
> approach is conceptually quite straight-forward, but has limits
> since it has to be pessimistic; i.e. a change in flink-core
> _must_ result in testing all modules.
> 3. Only run smoke tests when PR is opened, run heavy tests on demand.
> * With the introduction of the ci-bot we now have significantly
> more options on how to handle PR builds. One option could be to
> only run basic tests when the PR is created (which may be only
> modified modules, or all unit tests, or another low-cost
> scheme), and then have a committer trigger other builds (full
> test run, e2e tests, etc...) on demand.
> 4. Move more tests into cron builds
> * The budget version of 3); move certain tests that are either
> expensive (like some runtime tests that take minutes) or in
> rarely modified modules (like gelly) into cron jobs.
> 5. Gradle
> * Gradle was brought up a few times for it's built-in support for
> differential builds; basically providing 2) without the overhead
> of maintaining additional scripts.
> * To date no PoC was provided that shows it working in our CI
> environment (i.e., handling splits & caching etc).
> * This is the most disruptive change by a fair margin, as it would
> affect the entire project, developers and potentially users (f
> they build from source).
> 6. CI service
> * Our current artifact caching setup on Travis is basically a
> hack; we're basically abusing the Travis cache, which is meant
> for long-term caching, to ship build artifacts across jobs. It's
> brittle at times due to timing/visibility issues and on branches
> the cleanup processes can interfere with running builds. It is
> also not as effective as it could be.
> * There are CI services that provide build artifact caching out of
> the box, which could be useful for us.
> * To date, no PoC for using another CI service has been provided.
>
>
