Hi, > Currently all of the workers are running within docker daemons, > so all of the images are cached once they are pulled to a docker > daemon. The `worker_preparation` takes only 3 seconds: > https://ci.ursalabs.org/#/builders/66/builds/2157
Great! > but the package > build scripts needs to be ported to either crossbow or ursabot or the > new gitlab CI. I can work on this when we decide our approach. > Here is a PR which passes `--runtime=nvidia` option to the docker run > command, thus making CUDA enabled tests possible on buildbot's > docker workers: https://github.com/ursa-labs/ursabot/pull/118 I want to use it for https://github.com/apache/arrow/pull/4735 > We can also maintain our buildbot configuration in apache/arrow > similarly, but with a more flexible python based DSL. I like this approach. Because source code and how to test it should be the same place. If source code is changed, how to test may be changed too. Thanks, -- kou In <CAHM19a6Gz=ovst5caxfuunu4jr3-sea3-tksrztibftod9l...@mail.gmail.com> "Re: [DISCUSS] Ongoing Travis CI service degradation" on Sun, 30 Jun 2019 21:46:42 +0200, Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > On Sun, Jun 30, 2019 at 12:03 AM Sutou Kouhei <k...@clear-code.com> wrote: > >> Hi, >> >> How about creating a mirror repository on >> https://gitlab.com/ only to run CI jobs? >> >> This is an idea that is described in >> https://issues.apache.org/jira/browse/ARROW-5673 . >> > I do agree that We should investigate the features provided by > GitLab. Buildbot might not be so familiar to others, so I'm trying > to provide some details to see how it compares to GitLab CI. > >> >> GitLab CI can attach external workers. So we can increase CI >> capacity by adding our new workers. GitLab also provides >> Docker registry. It means that we can cache built Docker >> images for our CI. It will reduce CI time. >> > Currently all of the workers are running within docker daemons, > so all of the images are cached once they are pulled to a docker > daemon. The `worker_preparation` takes only 3 seconds: > https://ci.ursalabs.org/#/builders/66/builds/2157 > >> >> The feature to create a mirror repository for CI isn't >> included in the Free tier on https://gitlab.com/ . But >> https://gitlab.com/ provides the Gold tier features to open >> source project: >> https://about.gitlab.com/solutions/github/#open-source-projects >> So we can use this feature. >> >> >> Here are advantages I think to use GitLab CI: >> >> * We can increase CI capacity by adding our new workers. >> * GitLab Runner (CI job runner) can work on GNU/Linux, macOS >> and Windows: https://docs.gitlab.com/runner/#requirements >> It means that we can increase CI capacity of all of them. >> > The same is true for buildbot, the workers are basically python twisted > applications, so we can host them on any platform. > >> >> * We can reduce CI time by caching built Docker images. >> * It will reduce package build job time especially. >> > As I mentioned above the same is true for buildbot, but the package > build scripts needs to be ported to either crossbow or ursabot or the > new gitlab CI. > Mentioning crossbow, I've recently added support for azure pipelines > and circleci. For the docker tests (which are running the docker-compose > commands) we could add a gitlab specific template yml to test gitlab's > docker capabilities, see the current templates at > https://github.com/apache/arrow/tree/master/dev/tasks/docker-tests > >> >> * We can run CUDA related tests in CI by adding CUDA >> enabled workers. >> > Here is a PR which passes `--runtime=nvidia` option to the docker run > command, thus making CUDA enabled tests possible on buildbot's > docker workers: https://github.com/ursa-labs/ursabot/pull/118 > >> >> * We can manage CI jobs in https://github.com/apache/arrow >> repository. >> * GitLab CI uses .gitlab-ci.yml like .travis.yml for >> Travis CI. >> > We can also maintain our buildbot configuration in apache/arrow > similarly, but with a more flexible python based DSL. > >> >> >> If we create a mirror repository for CI on >> https://gitlab.com/ , https://gitlab.com/ursa-labs/arrow >> will be a good URL. >> >> >> Thanks, >> -- >> kou >> >> In <CAJPUwMBpxzCLZEM2=wwb+necdwz0z0kjd5egss3uqoiirk-...@mail.gmail.com> >> "Re: [DISCUSS] Ongoing Travis CI service degradation" on Sat, 29 Jun >> 2019 14:54:19 -0500, >> Wes McKinney <wesmck...@gmail.com> wrote: >> >> > hi Rok, >> > >> > I would guess that GitHub Actions will have the same resource and >> > hardware limitations that Travis CI and Appveyor currently have, as >> > well as organization-level resource contention with the rest of the >> > ASF. >> > >> > We need to have dedicated, powerful hardware (more cores, more RAM), >> > with more capabilities (architectures other than x86, and with GPUs), >> > that can run jobs longer than 50 minutes, with the ability to scale up >> > as the project grows in # of contributions per month. In the past >> > month Arrow had 4300 hours of builds on Travis CI. What will happen >> > when we need 10,000 or more hours per month to verify all of our >> > patches? At the current rapid rate of project growth it is only a >> > matter of time. >> > >> > I made a graph of commits to master by month: >> > >> > https://imgur.com/a/02TtGXx >> > >> > With nearly ~300 commits in the month of June alone, it begs the >> > question how to support 500 commits per month, or 1000. >> > >> > - Wes >> > >> > >> > >> > On Sat, Jun 29, 2019 at 5:19 AM Rok Mihevc <rok.mih...@gmail.com> wrote: >> >> >> >> GitHub Actions are currently in limited public beta and appear to be >> >> similar to GitLab CI: https://github.com/features/actions >> >> More here: https://help.github.com/en/articles/about-github-actions >> >> >> >> Rok >> >> >> >> On Fri, Jun 28, 2019 at 7:06 PM Wes McKinney <wesmck...@gmail.com> >> wrote: >> >> >> >> > Based on the discussion in >> >> > https://issues.apache.org/jira/browse/INFRA-18533 it does not appear >> >> > to be ASF Infra's inclination to allow projects to donate money to the >> >> > Foundation to get more build resources on Travis CI. Our likely only >> >> > solution is going to be to reduce our dependence on Travis CI. In the >> >> > short term, I would say that the sooner we can migrate all of our >> >> > Linux builds to docker-compose form to aid in this transition, the >> >> > better >> >> > >> >> > We are hiring in our organization (Ursa Labs) for a dedicated role to >> >> > support CI and development lifecycle automation (packaging, >> >> > benchmarking, releases, etc.) in the Apache Arrow project, so I hope >> >> > that we can provide even more help to resolve these issues in the >> >> > future than we already are >> >> > >> >> > On Wed, Jun 26, 2019 at 11:35 AM Antoine Pitrou <anto...@python.org> >> >> > wrote: >> >> > > >> >> > > >> >> > > Also note that the situation with AppVeyor isn't much better. >> >> > > >> >> > > Any "free as in beer" CI service is probably too capacity-limited >> for >> >> > > our needs now, unless it allows private workers (which apparently >> Gitlab >> >> > > CI does). >> >> > > >> >> > > Regards >> >> > > >> >> > > Antoine. >> >> > > >> >> > > >> >> > > Le 26/06/2019 à 18:32, Wes McKinney a écrit : >> >> > > > It seems that there is intermittent Apache-wide degradation of >> Travis >> >> > > > CI services -- I was looking at https://travis-ci.org/apache >> today and >> >> > > > there appeared to be a stretch of 3-4 hours where no queued >> builds on >> >> > > > github.com/apache were running at all. I initially thought that >> the >> >> > > > issue was contention with other Apache projects but even with >> >> > > > round-robin allocation and a concurrency limit (e.g. no Apache >> project >> >> > > > having more than 5-6 concurrent builds) that wouldn't explain why >> NO >> >> > > > builds are running. >> >> > > > >> >> > > > This is obviously disturbing given how reliant we are on Travis >> CI to >> >> > > > validate patches to be merged. >> >> > > > >> >> > > > I've opened a support ticket with Travis CI to see if they can >> provide >> >> > > > some insight into what's going on. There is also an INFRA ticket >> where >> >> > > > other projects have reported some similar experiences >> >> > > > >> >> > > > https://issues.apache.org/jira/browse/INFRA-18533 >> >> > > > >> >> > > > As a meta-comment, at some point Apache Arrow is going to need to >> move >> >> > > > off of public CI services for patch validation so that we can have >> >> > > > unilateral control over scaling our build / test resources as the >> >> > > > community grows larger. As the most active merger of patches (I >> have >> >> > > > merged over 50% of pull requests over the project's history) this >> >> > > > affects me greatly as I am often monitoring builds on many open >> PRs so >> >> > > > that I can merge them as soon as possible. We are often resorting >> to >> >> > > > builds on contributor's forks (assuming they have enabled Travis >> CI / >> >> > > > Appveyor) >> >> > > > >> >> > > > As some context around Travis CI in particular, in January Travis >> CI >> >> > > > was acquired by Idera, a private equity (I think?) developer tools >> >> > > > conglomerate. It's likely that we're seeing some "maximize profit, >> >> > > > minimize costs" behavior in play, so the recent experience could >> >> > > > become the new normal. >> >> > > > >> >> > > > - Wes >> >> > > > >> >> > >>
