Hi all,

I wanted to understand the impact of the hardware we are using for running
our tests. Each travis worker has 2 virtual cores, and 7.5 gb memory [1].
They are using Google Cloud Compute Engine *n1-standard-2* instances.
Running a full "mvn clean verify" takes *03:32 h* on such a machine type.

Running the same workload on a 32 virtual cores, 64 gb machine, takes *1:21
h*.

What is interesting are the per-module build time differences.
Modules which are parallelizing tests well greatly benefit from the
additional cores:
"flink-tests" 36:51 min vs 4:33 min
"flink-runtime" 23:41 min vs 3:47 min
"flink-table-planner" 15:54 min vs 3:13 min

On the other hand, we have modules which are not parallel at all:
"flink-connector-kafka": 16:32 min vs 15:19 min
"flink-connector-kafka-0.11": 9:52 min vs 7:46 min
Also, the checkstyle plugin is not scaling at all.

Chesnay reported some significant speedups by reusing forks.
I don't know how much effort it would be to make the Kafka tests
parallelizable. In total, they currently use 30 minutes on the big machine
(while 31 CPUs are idling :) )

Let me know what you think about these results. If the community is
generally interested in further investigating into that direction, I could
look into software to orchestrate this, as well as sponsors for such an
infrastructure.

[1] https://docs.travis-ci.com/user/reference/overview/


On Fri, Aug 16, 2019 at 3:27 PM Chesnay Schepler <ches...@apache.org> wrote:

> @Aljoscha Shading takes a few minutes for a full build; you can see this
> quite easily by looking at the compile step in the misc profile
> <https://api.travis-ci.org/v3/job/572560060/log.txt>; all modules that
> longer than a fraction of a section are usually caused by shading lots
> of classes. Note that I cannot tell you how much of this is spent on
> relocations, and how much on writing the jar.
>
> Personally, I'd very much like us to move all shading to flink-shaded;
> this would finally allows us to use newer maven versions without needing
> cumbersome workarounds for flink-dist. However, this isn't a trivial
> affair in some cases; IIRC calcite could be difficult to handle.
>
> On another note, this would also simplify switching the main repo to
> another build system, since you would no longer had to deal with
> relocations, just packaging + merging NOTICE files.
>
> @BowenLi I disagree, flink-shaded does not include any tests,  API
> compatibility checks, checkstyle, layered shading (e.g., flink-runtime
> and flink-dist, where both relocate dependencies and one is bundled by
> the other), and, most importantly, CI (and really, without CI being
> covered in a PoC there's nothing to discuss).
>
> On 16/08/2019 15:13, Aljoscha Krettek wrote:
> > Speaking of flink-shaded, do we have any idea what the impact of shading
> is on the build time? We could get rid of shading completely in the Flink
> main repository by moving everything that we shade to flink-shaded.
> >
> > Aljoscha
> >
> >> On 16. Aug 2019, at 14:58, Bowen Li <bowenl...@gmail.com> wrote:
> >>
> >> +1 to Till's points on #2 and #5, especially the potential
> non-disruptive,
> >> gradual migration approach if we decide to go that route.
> >>
> >> To add on, I want to point it out that we can actually start with
> >> flink-shaded project [1] which is a perfect candidate for PoC. It's of
> much
> >> smaller size, totally isolated from and not interfered with flink
> project
> >> [2], and it actually covers most of our practical feature requirements
> for
> >> a build tool - all making it an ideal experimental field.
> >>
> >> [1] https://github.com/apache/flink-shaded
> >> [2] https://github.com/apache/flink
> >>
> >>
> >> On Fri, Aug 16, 2019 at 4:52 AM Till Rohrmann <trohrm...@apache.org>
> wrote:
> >>
> >>> For the sake of keeping the discussion focused and not cluttering the
> >>> discussion thread I would suggest to split the detailed reporting for
> >>> reusing JVMs to a separate thread and cross linking it from here.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Fri, Aug 16, 2019 at 1:36 PM Chesnay Schepler <ches...@apache.org>
> >>> wrote:
> >>>
> >>>> Update:
> >>>>
> >>>> TL;DR: table-planner is a good candidate for enabling fork reuse right
> >>>> away, while flink-tests has the potential for huge savings, but we
> have
> >>>> to figure out some issues first.
> >>>>
> >>>>
> >>>> Build link: https://travis-ci.org/zentol/flink/builds/572659220
> >>>>
> >>>> 4/8 profiles failed.
> >>>>
> >>>> No speedup in libraries, python, blink_planner, 7 minutes saved in
> >>>> libraries (table-planner).
> >>>>
> >>>> The kafka and connectors profiles both fail in kafka tests due to
> >>>> producer leaks, and no speed up could be confirmed so far:
> >>>>
> >>>> java.lang.AssertionError: Detected producer leak. Thread name:
> >>>> kafka-producer-network-thread | producer-239
> >>>>         at org.junit.Assert.fail(Assert.java:88)
> >>>>         at
> >>>>
> >>>
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011ITCase.checkProducerLeak(FlinkKafkaProducer011ITCase.java:677)
> >>>>         at
> >>>>
> >>>
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011ITCase.testFlinkKafkaProducer011FailBeforeNotify(FlinkKafkaProducer011ITCase.java:210)
> >>>>
> >>>> The tests profile failed due to various errors in migration tests:
> >>>>
> >>>> junit.framework.AssertionFailedError: Did not see the expected
> >>> accumulator
> >>>> results within time limit.
> >>>>         at
> >>>>
> >>>
> org.apache.flink.test.migration.TypeSerializerSnapshotMigrationITCase.testSavepoint(TypeSerializerSnapshotMigrationITCase.java:141)
> >>>> *However*, a normal tests run takes 40 minutes, while this one above
> >>>> failed after 19 minutes and is only missing the migration tests (which
> >>>> currently need 6-7 minutes). So we could save somewhere between 15 to
> 20
> >>>> minutes here.
> >>>>
> >>>>
> >>>> Finally, the misc profiles fails in YARN:
> >>>>
> >>>> java.lang.AssertionError
> >>>>         at org.apache.flink.yarn.YARNITCase.setup(YARNITCase.java:64)
> >>>>
> >>>> No significant speedup could be observed in other modules; for
> >>>> flink-yarn-tests we can maybe get a minute or 2 out of it.
> >>>>
> >>>> On 16/08/2019 10:43, Chesnay Schepler wrote:
> >>>>> There appears to be a general agreement that 1) should be looked
> into;
> >>>>> I've setup a branch with fork reuse being enabled for all tests; will
> >>>>> report back the results.
> >>>>>
> >>>>> On 15/08/2019 09:38, Chesnay Schepler wrote:
> >>>>>> Hello everyone,
> >>>>>>
> >>>>>> improving our build times is a hot topic at the moment so let's
> >>>>>> discuss the different ways how they could be reduced.
> >>>>>>
> >>>>>>
> >>>>>>        Current state:
> >>>>>>
> >>>>>> First up, let's look at some numbers:
> >>>>>>
> >>>>>> 1 full build currently consumes 5h of build time total ("total
> >>>>>> time"), and in the ideal case takes about 1h20m ("run time") to
> >>>>>> complete from start to finish. The run time may fluctuate of course
> >>>>>> depending on the current Travis load. This applies both to builds on
> >>>>>> the Apache and flink-ci Travis.
> >>>>>>
> >>>>>> At the time of writing, the current queue time for PR jobs
> (reminder:
> >>>>>> running on flink-ci) is about 30 minutes (which basically means that
> >>>>>> we are processing builds at the rate that they come in), however we
> >>>>>> are in an admittedly quiet period right now.
> >>>>>> 2 weeks ago the queue times on flink-ci peaked at around 5-6h as
> >>>>>> everyone was scrambling to get their changes merged in time for the
> >>>>>> feature freeze.
> >>>>>>
> >>>>>> (Note: Recently optimizations where added to ci-bot where pending
> >>>>>> builds are canceled if a new commit was pushed to the PR or the PR
> >>>>>> was closed, which should prove especially useful during the rush
> >>>>>> hours we see before feature-freezes.)
> >>>>>>
> >>>>>>
> >>>>>>        Past approaches
> >>>>>>
> >>>>>> Over the years we have done rather few things to improve this
> >>>>>> situation (hence our current predicament).
> >>>>>>
> >>>>>> Beyond the sporadic speedup of some tests, the only notable
> reduction
> >>>>>> in total build times was the introduction of cron jobs, which
> >>>>>> consolidated the per-commit matrix from 4 configurations (different
> >>>>>> scala/hadoop versions) to 1.
> >>>>>>
> >>>>>> The separation into multiple build profiles was only a work-around
> >>>>>> for the 50m limit on Travis. Running tests in parallel has the
> >>>>>> obvious potential of reducing run time, but we're currently hitting
> a
> >>>>>> hard limit since a few modules (flink-tests, flink-runtime,
> >>>>>> flink-table-planner-blink) are so loaded with tests that they nearly
> >>>>>> consume an entire profile by themselves (and thus no further
> >>>>>> splitting is possible).
> >>>>>>
> >>>>>> The rework that introduced stages, at the time of introduction, did
> >>>>>> also not provide a speed up, although this changed slightly once
> more
> >>>>>> profiles were added and some optimizations to the caching have been
> >>>>>> made.
> >>>>>>
> >>>>>> Very recently we modified the surefire-plugin configuration for
> >>>>>> flink-table-planner-blink to reuse JVM forks for IT cases, providing
> >>>>>> a significant speedup (18 minutes!). So far we have not seen any
> >>>>>> negative consequences.
> >>>>>>
> >>>>>>
> >>>>>>        Suggestions
> >>>>>>
> >>>>>> This is a list of /all /suggestions for reducing run/total times
> that
> >>>>>> I have seen recently (in other words, they aren't necessarily mine
> >>>>>> nor may I agree with all of them).
> >>>>>>
> >>>>>> 1. Enable JVM reuse for IT cases in more modules.
> >>>>>>      * We've seen significant speedups in the blink planner, and
> this
> >>>>>>        should be applicable for all modules. However, I presume
> >>> there's
> >>>>>>        a reason why we disabled JVM reuse (information on this would
> >>> be
> >>>>>>        appreciated)
> >>>>>> 2. Custom differential build scripts
> >>>>>>      * Setup custom scripts for determining which modules might be
> >>>>>>        affected by change, and manipulate the splits accordingly.
> This
> >>>>>>        approach is conceptually quite straight-forward, but has
> limits
> >>>>>>        since it has to be pessimistic; i.e. a change in flink-core
> >>>>>>        _must_ result in testing all modules.
> >>>>>> 3. Only run smoke tests when PR is opened, run heavy tests on
> demand.
> >>>>>>      * With the introduction of the ci-bot we now have significantly
> >>>>>>        more options on how to handle PR builds. One option could be
> to
> >>>>>>        only run basic tests when the PR is created (which may be
> only
> >>>>>>        modified modules, or all unit tests, or another low-cost
> >>>>>>        scheme), and then have a committer trigger other builds (full
> >>>>>>        test run, e2e tests, etc...) on demand.
> >>>>>> 4. Move more tests into cron builds
> >>>>>>      * The budget version of 3); move certain tests that are either
> >>>>>>        expensive (like some runtime tests that take minutes) or in
> >>>>>>        rarely modified modules (like gelly) into cron jobs.
> >>>>>> 5. Gradle
> >>>>>>      * Gradle was brought up a few times for it's built-in support
> for
> >>>>>>        differential builds; basically providing 2) without the
> >>> overhead
> >>>>>>        of maintaining additional scripts.
> >>>>>>      * To date no PoC was provided that shows it working in our CI
> >>>>>>        environment (i.e., handling splits & caching etc).
> >>>>>>      * This is the most disruptive change by a fair margin, as it
> >>> would
> >>>>>>        affect the entire project, developers and potentially users
> (f
> >>>>>>        they build from source).
> >>>>>> 6. CI service
> >>>>>>      * Our current artifact caching setup on Travis is basically a
> >>>>>>        hack; we're basically abusing the Travis cache, which is
> meant
> >>>>>>        for long-term caching, to ship build artifacts across jobs.
> >>> It's
> >>>>>>        brittle at times due to timing/visibility issues and on
> >>> branches
> >>>>>>        the cleanup processes can interfere with running builds. It
> is
> >>>>>>        also not as effective as it could be.
> >>>>>>      * There are CI services that provide build artifact caching out
> >>> of
> >>>>>>        the box, which could be useful for us.
> >>>>>>      * To date, no PoC for using another CI service has been
> provided.
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >
>
>

Reply via email to