Thanks Robert for the updates! And thanks a lot for all the efforts to investigate, experiment and tune Azure Pipelines for Flink building. Big +1 for it.
It would be great that the community building can be extended with custom machines so that the tests would not be queued for long with daily growing PRs. The increased timeout would be also very helpful. The 50min timeout for free travis accounts is a pain currently, especially when we'd like to run e2e tests in our own travis. And I had to manually split the jobs to make it possible to pass. Thanks, Zhu Zhu Robert Metzger <rmetz...@apache.org> 于2019年12月4日周三 下午6:36写道： > Hi all, > > as a follow up from our discussion on reducing the build time [1], I would > like to propose migrating our build infrastructure to Azure Pipelines (away > from Travis). > > I believe that we have reached the limits of what Travis can provide the > Flink community, and I don't want the build system to limit or influence > the project's growth. > > *Benefits:* > 1. The free Travis account are limited to 5 parallel builds, with a timeout > of 50 minutes. Azure offers *10 parallel builds with 300 minute timeouts > *for > free for open source projects. > 2. Azure Pipelines allows us to *add custom build machines* to the pool of > 10 free parallel builders. > This will allow the Flink community to scale the available build capacity > as the project grows. We are dependent on donations from supporting > companies, but I believe that it is easier for companies to donate machines > than money. > Alibaba is willing to provide 10 machines, with 32 cores each to the Flink > project for this purpose. > In addition, Xiyuan, who's working on adding ARM support for Flink provided > me with 2 ARM machines (16 cores each). > I want to use the custom, more efficient build machines for building > Flink's pull requests and master-pushes. > 3. *Azure Pipelines is a more feature-rich tool*, allowing for example to > transfer intermediate build artifacts between pipeline stages. This will > allow us to make the build more reliable (we are currently abusing the > caching mechanism in Travis for this). > It also has some basic analytics on test results / flaky tests etc. > > *Known problems:* > - Initially, we might see different build instabilities than before > - There's a higher maintenance overhead for the custom build machines > (keeping them up to date etc.) > - We can not use the build status integration of AZP, because they require > write access to the repository's source. The foundation does not allow that > [2]. > I propose to extend flinkbot / the flink-ci repository. > > *Current Status:* > - I'm able [3] to execute [4] the current custom build scripts on Azure > Pipelines: This means that we will have one compile stage, and N testing > jobs in the 2nd stage. Currently, we have N=10 testing jobs. > The time from the start of a build till all tests have completed is 1h22 > minutes. > - I'm working on getting the nightly end to end tests to run on the new > infrastructure. > - I'm working on getting the build to work on our pool of custom machines > as well > - I'm working on setting up the full matrix of builds (different scala, > hadoop etc. versions) for the nightlies > > *Next Steps:* > - I propose to document the entire build system in the Flink Wiki > - Once Azure can cover the same pull request tests as Travis, I would set > it up to run in parallel (including Flinkbot posting links to Azure). I > hope that this phase lasts for 1-2 weeks only, so that we do not have to > maintain things concurrently. I will monitor the build stability closely, > but would expect some support with debugging potential issues from the > contributors. > - Once there are no problems with the new setup, we remove the Travis > setup. > - Independently, I will work on triggering builds from master / release - > branch pushes, as well as cron builds from the master branch ... all this > will be described in the Wiki. > > > *Timeline:*- Once I have the feeling that people are supportive of the > idea, I will start documenting in the Wiki. The first pull requests should > show up after a few more days. > I will do a one month parental leave starting some time later in December, > which will probably delay things a bit. I hope to have everything finished > by end of January. > > I'm happy to hear your thoughts on this work. > If nobody objects, I will start documenting the system and prepare > everything for the migration. > > Best, > Robert > > > > [1] > > https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E > [2] https://issues.apache.org/jira/browse/INFRA-17030 > [3] https://github.com/rmetzger/flink/tree/azure_playground > [4] https://dev.azure.com/rmetzger/Flink/_build?definitionId=4&_a=summary >
