Yes, we can ensure the same (or better) experience for contributors. On the powerful machines, builds finish in 1.5 hours (without any caching enabled).
Azure Pipelines offers 10 concurrent builds with a timeout of 6 hours for a build for open source projects. Flink needs 3.5 hours on that infra (not parallelized at all, no caching). These free machines are very similar to those of Travis, so I expect no build time regressions, if we set it up similarly. On Wed, Sep 4, 2019 at 9:19 AM Chesnay Schepler <ches...@apache.org> wrote: > Will using more powerful for the project make it more difficult to > ensure that contributor builds are still running in a reasonable time? > > As an example of this happening on Travis, contributors currently cannot > run all e2e tests since they timeout, but on apache we have a larger > timeout. > > On 03/09/2019 18:57, Robert Metzger wrote: > > Hi all, > > > > I wanted to give a short update on this: > > - Arvid, Aljoscha and I have started working on a Gradle PoC, currently > > working on making all modules compile and test with Gradle. We've also > > identified some problematic areas (shading being the most obvious one) > > which we will analyse as part of the PoC. > > The goal is to see how much Gradle helps to parallelise our build, and to > > avoid duplicate work (incremental builds). > > > > - I am working on setting up a Flink testing infrastructure based on > Azure > > Pipelines, using more powerful hardware. Alibaba kindly provided me with > > two 32 core machines (temporarily), and another company reached out to > > privately, looking into options for cheap, fast machines :) > > If nobody in the community disagrees, I am going to set up Azure > Pipelines > > with our apache/flink GitHub as a build infrastructure that exists next > to > > Flinkbot and flink-ci. I would like to make sure that Azure Pipelines is > > equally or even more reliable than Travis, and I want to see what the > > required maintenance work is. > > On top of that, Azure Pipelines is a very feature-rich tool with a lot of > > nice options for us to improve the build experience (statistics about > tests > > (flaky tests etc.), nice docker support, plenty of free build resources > for > > open source projects, ...) > > > > Best, > > Robert > > > > > > > > > > > > On Mon, Aug 19, 2019 at 5:12 PM Robert Metzger <rmetz...@apache.org> > wrote: > > > >> Hi all, > >> > >> I have summarized all arguments mentioned so far + some additional > >> research into a Wiki page here: > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125309279 > >> > >> I'm happy to hear further comments on my summary! I'm pretty sure we can > >> find more pro's and con's for the different options. > >> > >> My opinion after looking at the options: > >> > >> - Flink relies on an outdated build tool (Maven), while a good > >> alternative is well-established (gradle), and will likely provide a > much > >> better CI and local build experience through incremental build and > cached > >> intermediates. > >> Scripting around Maven, or splitting modules / test execution / > >> repositories won't solve this problem. We should rather spend the > effort in > >> migrating to a modern build tool which will provide us benefits in > the long > >> run. > >> - Flink relies on a fairly slow build service (Travis CI), while > >> simply putting more money onto the problem could cut the build time > at > >> least in half. > >> We should consider using a build service that provides bigger > machines > >> to solve our build time problem. > >> > >> My opinion is based on many assumptions (gradle is actually as fast as > >> promised (haven't used it before), we can build Flink with gradle, we > find > >> sponsors for bigger build machines) that we need to test first through > PoCs. > >> > >> Best, > >> Robert > >> > >> > >> > >> > >> On Mon, Aug 19, 2019 at 10:26 AM Aljoscha Krettek <aljos...@apache.org> > >> wrote: > >> > >>> I did a quick test: a normal "mvn clean install -DskipTests > >>> -Drat.skip=true -Dmaven.javadoc.skip=true -Punsafe-mapr-repo” on my > machine > >>> takes about 14 minutes. After removing all mentions of > maven-shade-plugin > >>> the build time goes down to roughly 11.5 minutes. (Obviously the > resulting > >>> Flink won’t work, because some expected stuff is not packaged and most > of > >>> the end-to-end tests use the shade plugin to package the jars for > testing. > >>> > >>> Aljoscha > >>> > >>>> On 18. Aug 2019, at 19:52, Robert Metzger <rmetz...@apache.org> > wrote: > >>>> > >>>> Hi all, > >>>> > >>>> I wanted to understand the impact of the hardware we are using for > >>> running > >>>> our tests. Each travis worker has 2 virtual cores, and 7.5 gb memory > >>> [1]. > >>>> They are using Google Cloud Compute Engine *n1-standard-2* instances. > >>>> Running a full "mvn clean verify" takes *03:32 h* on such a machine > >>> type. > >>>> Running the same workload on a 32 virtual cores, 64 gb machine, takes > >>> *1:21 > >>>> h*. > >>>> > >>>> What is interesting are the per-module build time differences. > >>>> Modules which are parallelizing tests well greatly benefit from the > >>>> additional cores: > >>>> "flink-tests" 36:51 min vs 4:33 min > >>>> "flink-runtime" 23:41 min vs 3:47 min > >>>> "flink-table-planner" 15:54 min vs 3:13 min > >>>> > >>>> On the other hand, we have modules which are not parallel at all: > >>>> "flink-connector-kafka": 16:32 min vs 15:19 min > >>>> "flink-connector-kafka-0.11": 9:52 min vs 7:46 min > >>>> Also, the checkstyle plugin is not scaling at all. > >>>> > >>>> Chesnay reported some significant speedups by reusing forks. > >>>> I don't know how much effort it would be to make the Kafka tests > >>>> parallelizable. In total, they currently use 30 minutes on the big > >>> machine > >>>> (while 31 CPUs are idling :) ) > >>>> > >>>> Let me know what you think about these results. If the community is > >>>> generally interested in further investigating into that direction, I > >>> could > >>>> look into software to orchestrate this, as well as sponsors for such > an > >>>> infrastructure. > >>>> > >>>> [1] https://docs.travis-ci.com/user/reference/overview/ > >>>> > >>>> > >>>> On Fri, Aug 16, 2019 at 3:27 PM Chesnay Schepler <ches...@apache.org> > >>> wrote: > >>>>> @Aljoscha Shading takes a few minutes for a full build; you can see > >>> this > >>>>> quite easily by looking at the compile step in the misc profile > >>>>> <https://api.travis-ci.org/v3/job/572560060/log.txt>; all modules > that > >>>>> longer than a fraction of a section are usually caused by shading > lots > >>>>> of classes. Note that I cannot tell you how much of this is spent on > >>>>> relocations, and how much on writing the jar. > >>>>> > >>>>> Personally, I'd very much like us to move all shading to > flink-shaded; > >>>>> this would finally allows us to use newer maven versions without > >>> needing > >>>>> cumbersome workarounds for flink-dist. However, this isn't a trivial > >>>>> affair in some cases; IIRC calcite could be difficult to handle. > >>>>> > >>>>> On another note, this would also simplify switching the main repo to > >>>>> another build system, since you would no longer had to deal with > >>>>> relocations, just packaging + merging NOTICE files. > >>>>> > >>>>> @BowenLi I disagree, flink-shaded does not include any tests, API > >>>>> compatibility checks, checkstyle, layered shading (e.g., > flink-runtime > >>>>> and flink-dist, where both relocate dependencies and one is bundled > by > >>>>> the other), and, most importantly, CI (and really, without CI being > >>>>> covered in a PoC there's nothing to discuss). > >>>>> > >>>>> On 16/08/2019 15:13, Aljoscha Krettek wrote: > >>>>>> Speaking of flink-shaded, do we have any idea what the impact of > >>> shading > >>>>> is on the build time? We could get rid of shading completely in the > >>> Flink > >>>>> main repository by moving everything that we shade to flink-shaded. > >>>>>> Aljoscha > >>>>>> > >>>>>>> On 16. Aug 2019, at 14:58, Bowen Li <bowenl...@gmail.com> wrote: > >>>>>>> > >>>>>>> +1 to Till's points on #2 and #5, especially the potential > >>>>> non-disruptive, > >>>>>>> gradual migration approach if we decide to go that route. > >>>>>>> > >>>>>>> To add on, I want to point it out that we can actually start with > >>>>>>> flink-shaded project [1] which is a perfect candidate for PoC. It's > >>> of > >>>>> much > >>>>>>> smaller size, totally isolated from and not interfered with flink > >>>>> project > >>>>>>> [2], and it actually covers most of our practical feature > >>> requirements > >>>>> for > >>>>>>> a build tool - all making it an ideal experimental field. > >>>>>>> > >>>>>>> [1] https://github.com/apache/flink-shaded > >>>>>>> [2] https://github.com/apache/flink > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Aug 16, 2019 at 4:52 AM Till Rohrmann < > trohrm...@apache.org> > >>>>> wrote: > >>>>>>>> For the sake of keeping the discussion focused and not cluttering > >>> the > >>>>>>>> discussion thread I would suggest to split the detailed reporting > >>> for > >>>>>>>> reusing JVMs to a separate thread and cross linking it from here. > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Till > >>>>>>>> > >>>>>>>> On Fri, Aug 16, 2019 at 1:36 PM Chesnay Schepler < > >>> ches...@apache.org> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Update: > >>>>>>>>> > >>>>>>>>> TL;DR: table-planner is a good candidate for enabling fork reuse > >>> right > >>>>>>>>> away, while flink-tests has the potential for huge savings, but > we > >>>>> have > >>>>>>>>> to figure out some issues first. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Build link: https://travis-ci.org/zentol/flink/builds/572659220 > >>>>>>>>> > >>>>>>>>> 4/8 profiles failed. > >>>>>>>>> > >>>>>>>>> No speedup in libraries, python, blink_planner, 7 minutes saved > in > >>>>>>>>> libraries (table-planner). > >>>>>>>>> > >>>>>>>>> The kafka and connectors profiles both fail in kafka tests due to > >>>>>>>>> producer leaks, and no speed up could be confirmed so far: > >>>>>>>>> > >>>>>>>>> java.lang.AssertionError: Detected producer leak. Thread name: > >>>>>>>>> kafka-producer-network-thread | producer-239 > >>>>>>>>> at org.junit.Assert.fail(Assert.java:88) > >>>>>>>>> at > >>>>>>>>> > >>> > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011ITCase.checkProducerLeak(FlinkKafkaProducer011ITCase.java:677) > >>>>>>>>> at > >>>>>>>>> > >>> > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011ITCase.testFlinkKafkaProducer011FailBeforeNotify(FlinkKafkaProducer011ITCase.java:210) > >>>>>>>>> The tests profile failed due to various errors in migration > tests: > >>>>>>>>> > >>>>>>>>> junit.framework.AssertionFailedError: Did not see the expected > >>>>>>>> accumulator > >>>>>>>>> results within time limit. > >>>>>>>>> at > >>>>>>>>> > >>> > org.apache.flink.test.migration.TypeSerializerSnapshotMigrationITCase.testSavepoint(TypeSerializerSnapshotMigrationITCase.java:141) > >>>>>>>>> *However*, a normal tests run takes 40 minutes, while this one > >>> above > >>>>>>>>> failed after 19 minutes and is only missing the migration tests > >>> (which > >>>>>>>>> currently need 6-7 minutes). So we could save somewhere between > 15 > >>> to > >>>>> 20 > >>>>>>>>> minutes here. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Finally, the misc profiles fails in YARN: > >>>>>>>>> > >>>>>>>>> java.lang.AssertionError > >>>>>>>>> at > >>> org.apache.flink.yarn.YARNITCase.setup(YARNITCase.java:64) > >>>>>>>>> No significant speedup could be observed in other modules; for > >>>>>>>>> flink-yarn-tests we can maybe get a minute or 2 out of it. > >>>>>>>>> > >>>>>>>>> On 16/08/2019 10:43, Chesnay Schepler wrote: > >>>>>>>>>> There appears to be a general agreement that 1) should be looked > >>>>> into; > >>>>>>>>>> I've setup a branch with fork reuse being enabled for all tests; > >>> will > >>>>>>>>>> report back the results. > >>>>>>>>>> > >>>>>>>>>> On 15/08/2019 09:38, Chesnay Schepler wrote: > >>>>>>>>>>> Hello everyone, > >>>>>>>>>>> > >>>>>>>>>>> improving our build times is a hot topic at the moment so let's > >>>>>>>>>>> discuss the different ways how they could be reduced. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Current state: > >>>>>>>>>>> > >>>>>>>>>>> First up, let's look at some numbers: > >>>>>>>>>>> > >>>>>>>>>>> 1 full build currently consumes 5h of build time total ("total > >>>>>>>>>>> time"), and in the ideal case takes about 1h20m ("run time") to > >>>>>>>>>>> complete from start to finish. The run time may fluctuate of > >>> course > >>>>>>>>>>> depending on the current Travis load. This applies both to > >>> builds on > >>>>>>>>>>> the Apache and flink-ci Travis. > >>>>>>>>>>> > >>>>>>>>>>> At the time of writing, the current queue time for PR jobs > >>>>> (reminder: > >>>>>>>>>>> running on flink-ci) is about 30 minutes (which basically means > >>> that > >>>>>>>>>>> we are processing builds at the rate that they come in), > however > >>> we > >>>>>>>>>>> are in an admittedly quiet period right now. > >>>>>>>>>>> 2 weeks ago the queue times on flink-ci peaked at around 5-6h > as > >>>>>>>>>>> everyone was scrambling to get their changes merged in time for > >>> the > >>>>>>>>>>> feature freeze. > >>>>>>>>>>> > >>>>>>>>>>> (Note: Recently optimizations where added to ci-bot where > pending > >>>>>>>>>>> builds are canceled if a new commit was pushed to the PR or the > >>> PR > >>>>>>>>>>> was closed, which should prove especially useful during the > rush > >>>>>>>>>>> hours we see before feature-freezes.) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Past approaches > >>>>>>>>>>> > >>>>>>>>>>> Over the years we have done rather few things to improve this > >>>>>>>>>>> situation (hence our current predicament). > >>>>>>>>>>> > >>>>>>>>>>> Beyond the sporadic speedup of some tests, the only notable > >>>>> reduction > >>>>>>>>>>> in total build times was the introduction of cron jobs, which > >>>>>>>>>>> consolidated the per-commit matrix from 4 configurations > >>> (different > >>>>>>>>>>> scala/hadoop versions) to 1. > >>>>>>>>>>> > >>>>>>>>>>> The separation into multiple build profiles was only a > >>> work-around > >>>>>>>>>>> for the 50m limit on Travis. Running tests in parallel has the > >>>>>>>>>>> obvious potential of reducing run time, but we're currently > >>> hitting > >>>>> a > >>>>>>>>>>> hard limit since a few modules (flink-tests, flink-runtime, > >>>>>>>>>>> flink-table-planner-blink) are so loaded with tests that they > >>> nearly > >>>>>>>>>>> consume an entire profile by themselves (and thus no further > >>>>>>>>>>> splitting is possible). > >>>>>>>>>>> > >>>>>>>>>>> The rework that introduced stages, at the time of introduction, > >>> did > >>>>>>>>>>> also not provide a speed up, although this changed slightly > once > >>>>> more > >>>>>>>>>>> profiles were added and some optimizations to the caching have > >>> been > >>>>>>>>>>> made. > >>>>>>>>>>> > >>>>>>>>>>> Very recently we modified the surefire-plugin configuration for > >>>>>>>>>>> flink-table-planner-blink to reuse JVM forks for IT cases, > >>> providing > >>>>>>>>>>> a significant speedup (18 minutes!). So far we have not seen > any > >>>>>>>>>>> negative consequences. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Suggestions > >>>>>>>>>>> > >>>>>>>>>>> This is a list of /all /suggestions for reducing run/total > times > >>>>> that > >>>>>>>>>>> I have seen recently (in other words, they aren't necessarily > >>> mine > >>>>>>>>>>> nor may I agree with all of them). > >>>>>>>>>>> > >>>>>>>>>>> 1. Enable JVM reuse for IT cases in more modules. > >>>>>>>>>>> * We've seen significant speedups in the blink planner, > and > >>>>> this > >>>>>>>>>>> should be applicable for all modules. However, I presume > >>>>>>>> there's > >>>>>>>>>>> a reason why we disabled JVM reuse (information on this > >>> would > >>>>>>>> be > >>>>>>>>>>> appreciated) > >>>>>>>>>>> 2. Custom differential build scripts > >>>>>>>>>>> * Setup custom scripts for determining which modules > might be > >>>>>>>>>>> affected by change, and manipulate the splits > accordingly. > >>>>> This > >>>>>>>>>>> approach is conceptually quite straight-forward, but has > >>>>> limits > >>>>>>>>>>> since it has to be pessimistic; i.e. a change in > flink-core > >>>>>>>>>>> _must_ result in testing all modules. > >>>>>>>>>>> 3. Only run smoke tests when PR is opened, run heavy tests on > >>>>> demand. > >>>>>>>>>>> * With the introduction of the ci-bot we now have > >>> significantly > >>>>>>>>>>> more options on how to handle PR builds. One option > could > >>> be > >>>>> to > >>>>>>>>>>> only run basic tests when the PR is created (which may > be > >>>>> only > >>>>>>>>>>> modified modules, or all unit tests, or another low-cost > >>>>>>>>>>> scheme), and then have a committer trigger other builds > >>> (full > >>>>>>>>>>> test run, e2e tests, etc...) on demand. > >>>>>>>>>>> 4. Move more tests into cron builds > >>>>>>>>>>> * The budget version of 3); move certain tests that are > >>> either > >>>>>>>>>>> expensive (like some runtime tests that take minutes) > or in > >>>>>>>>>>> rarely modified modules (like gelly) into cron jobs. > >>>>>>>>>>> 5. Gradle > >>>>>>>>>>> * Gradle was brought up a few times for it's built-in > support > >>>>> for > >>>>>>>>>>> differential builds; basically providing 2) without the > >>>>>>>> overhead > >>>>>>>>>>> of maintaining additional scripts. > >>>>>>>>>>> * To date no PoC was provided that shows it working in > our CI > >>>>>>>>>>> environment (i.e., handling splits & caching etc). > >>>>>>>>>>> * This is the most disruptive change by a fair margin, as > it > >>>>>>>> would > >>>>>>>>>>> affect the entire project, developers and potentially > users > >>>>> (f > >>>>>>>>>>> they build from source). > >>>>>>>>>>> 6. CI service > >>>>>>>>>>> * Our current artifact caching setup on Travis is > basically a > >>>>>>>>>>> hack; we're basically abusing the Travis cache, which is > >>>>> meant > >>>>>>>>>>> for long-term caching, to ship build artifacts across > jobs. > >>>>>>>> It's > >>>>>>>>>>> brittle at times due to timing/visibility issues and on > >>>>>>>> branches > >>>>>>>>>>> the cleanup processes can interfere with running > builds. It > >>>>> is > >>>>>>>>>>> also not as effective as it could be. > >>>>>>>>>>> * There are CI services that provide build artifact > caching > >>> out > >>>>>>>> of > >>>>>>>>>>> the box, which could be useful for us. > >>>>>>>>>>> * To date, no PoC for using another CI service has been > >>>>> provided. > >>>>>>>>>>> > >>>>> > >>> > >
