On Sun, Jun 30, 2019 at 3:03 PM Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > > On Sun, Jun 30, 2019 at 9:12 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > I've justed created a parent JIRA for Docker-ifying all of the Linux > > builds in Travis CI > > > > https://issues.apache.org/jira/browse/ARROW-5801 > > > > I did Java here since it was one of the easier ones > > > > https://github.com/apache/arrow/pull/4761 > > I've written most of the Dockerfiles in the arrow repository and > writing hierarchical images (without native docker support for it) > and maintaining them is really painful, error prone and hardly testable. > > So I've started to write a small tool in order to overcome these > limitations, see https://github.com/ursa-labs/ursabot#define-docker-images >
This tool looks cool. What do you think about contributing this tool to Apache Arrow so that we can use it to generate some of our Dockerfiles? > > > > > > Expensive Docker images can be pushed to @ursalab on Docker Hub > > (https://cloud.docker.com/u/ursalab/repository/list) -- I will be > > happy to give any Arrow committer access to this Docker Hub > > organization to help maintain the images in the short term. > > > These docker images are built and pushed automatically by the tool > described above. Here you can find the DSL used for those > images: > https://github.com/ursa-labs/ursabot/blob/master/ursabot/docker.py#L372-L540 > > > > > Some of the others will require more work. > > > > We should think about how to refactor the build scripts for macOS in a > > way that is decoupled from Travis CI environment variables and custom > > build image details so they can more easily be run on arbitrary macOS > > build workers. > > > > On Sun, Jun 30, 2019 at 11:26 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > > GitLab is currently more mature but on the other hand we're already on > > > > GitHub. We should probably evaluate both options if we go this way. > > > > > > We have to keep the code repository on GitHub because all Apache > > > projects are on GitHub now. How projects manage patches and CI is up > > > to each project, though. Other projects I'm familiar with like Apache > > > Kudu and Apache Impala use Gerrit and Jenkins for their code review > > > and CI, respectively. > > > > > > If we can use GitLab CI and get it to make status reports into our PR > > > queue on GitHub, that would be nice. Buildbot is another option > > > (though they are not mutually exclusive). My colleagues and I plan go > > > to continue investing in our Buildbot infrastructure though not > > > necessarily to the exclusion of other things. > > > > > > Perhaps we can set up a proof of concept on GitLab to see what things > > > look like. I've set up a repository mirror at > > > > > > https://gitlab.com/ursa-labs/arrow > > > > > > where we can experiment > > > > > > On Sat, Jun 29, 2019 at 10:02 PM Jed Brown <j...@jedbrown.org> wrote: > > > > > > > > Sutou Kouhei <k...@clear-code.com> writes: > > > > > > > > > How about creating a mirror repository on > > > > > https://gitlab.com/ only to run CI jobs? > > > > > > > > > > This is an idea that is described in > > > > > https://issues.apache.org/jira/browse/ARROW-5673 . > > > > > > > > > > GitLab CI can attach external workers. So we can increase CI > > > > > capacity by adding our new workers. GitLab also provides > > > > > Docker registry. It means that we can cache built Docker > > > > > images for our CI. It will reduce CI time. > > > > > > > > I have some experience with GitLab-CI. The gitlab-runner is great and > > > > easy to deploy. GitLab-CI integrates structured artifacts (like which > > > > tests pass and their output [1] and changes to continuous metrics like > > > > performance [2]) very nicely into GitLab Merge Requests, but when you > > > > connect to an external repository (GitHub, etc.), it only reports > > > > pass/fail and you can't access the structured artifacts [3], only the > > > > console logs and compressed archives of artifacts if you use that > > > > feature. > > > > > > > > If you're happy with clicking through to console logs in case a > > pipeline > > > > fails (the Travis model), then GitLab-CI is easy to use and will serve > > > > your purposes. If you really want the structured features, then I'd > > > > encourage you to mention that in [3]. > > > > > > > > [1] https://docs.gitlab.com/ee/ci/junit_test_reports.html#how-it-works > > > > [2] > > https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html#how-it-works > > > > [3] https://gitlab.com/gitlab-org/gitlab-ce/issues/60158 > >
