Hi, A straightforward way would be to run non-critical CI jobs as nightlies. Nightly package builds work pretty well, see the following link https://github.com/kszucs/crossbow/branches/all?query=nightly the notification logic requires improvement though. We should also run integrations tests regularly (dask, hdfs, spark). If We use a non apache queue repository for crossbow, We could even submit jobs to other CI services (e.g. CircleCI) to increase the build parallelism.
Krisztian On Aug 6 2018, at 1:23 pm, Antoine Pitrou <anto...@python.org> wrote: > > > Hello, > Our CI jobs are taking longer and longer. The main reason seem not to > be that our test suites become more thorough (running tests actually > seems to account for a very minor fraction of CI times) but the combined > fact that 1) fetching dependencies and building is slow 2) we have many > configurations tested on our CI jobs. > > The slowest job on our Travis-CI configuration routinely takes more 40 > minutes (this is for a single job, where everything is essentially > serialized). As for AppVeyor, the different jobs in a build there are > mostly serialized, which balloons the total build time. > > If we ever move parquet-cpp into the Arrow repo, this will probably > become worse again (though it might be better for parquet-cpp itself, > which doesn't seem to have received a lot of care on the CI side: for > instance, parquet-cpp AppVeyor builds are abysmally slow). > > I think we will have to cut down on the number of things we exercise on > the per-build CI builds. This can be done along two axes: > 1) remove some build steps that we deem non-critical or even unimportant > 2) remove some build configurations entirely (for instance I don't > understand why we need a "static CRT" build on Windows, or worse, why we > need to have NMake-based builds at all) > > Any thoughts? > Regards > Antoine.
