On Wed, 2018-08-08 at 17:40 +0100, Daniel Stone wrote: > Hi Juan, > > On Wed, 8 Aug 2018 at 16:45, Juan A. Suarez Romero <jasua...@igalia.com> > wrote: > > This is a first part of a more complete proposal to use GitLab CI to build > > and > > test Mesa. This first part just adds the required pieces to build Mesa, > > using the > > different supported tools (meson, autotools, and scons). > > This is great - I'm super excited to see it happen! > > > Unfortunately, Rocker is a tool that is not maintained anymore > > (https://github.com/grammarly/rocker). We still use it because it provides > > everything we need, and fortunately we don't need any more features. > > > > Maybe there are other alternatives out there, but we were happy with this > > and > > hence our proposal. If we want to use Docker rather than Rocker, then we > > could > > use template tools like Jinja, and forget about caching during build, which > > will > > impact on the build time. > > This bit is a bit concerning, especially since it makes CI harder to > approach for people who might want to work on it. There are quite a > few alternate tools (buildah comes to mind, as well as umoci/skopeo) > which might end up being better, particularly for scriptability. But > I'm not volunteering to do that work, so take this with a grain of > salt! >
Yes, I knew whis could be a concerning point. There are two points to cover: - Dockerfile templates, to allow complex building from single Dockerfile - Allow external mount on Docker build buildah indeed provides a way for the later. Still, we need to fix the the former, unless we drop the Dockerfile and use whatever config syntax uses buildah. Otherwise, something like a Jinja template or similar, and proper tools, could fix the problem. One of the things we decided to continue, and keep using Rocker, is that this tool fixes both problems at the same time. And the syntax of the Rockerfiles is the same as the Dockerfiles, but with more options. So anyone familiarized with Docker will have no problems with the Rockerfiles. A final reason for Rocker is how easy is to install: it is a single executable file. This means that if someone needs to build the containers locally, they only need to download a single file and build the containers, no matter the distro they are using. Anyway, I'm not paid by Rocker :). So I'm fine if we need to search for an alternative. > > ## Involved stages > > > > The dependencies required to build Mesa doesn't change very frequently, so > > building them everytime is a waste of time. As Docker allows to create > > images > > based on the content of other images, we have defined the setup in several > > stages. > > > > On a first stage a "base" image is built. This image contains almost all the > > dependencies required to build Mesa. Worth to mention that libdrm is > > excluded > > here, as this is a dependency that really changes quite frequently, so we > > postpone the installation for further stages. > > > > One we have the "base" image, we create different images with the different > > LLVM > > compilers. This ensure that when using a specific image we only have that > > LLVM > > version, and not any other. > > > > An important point here is that, these builts appears in the pipeline, they > > are > > not actually built if not required. That is, in the case of the base image, > > if > > the Rockerfile used to create the image has changed with respect to the one > > used > > to create the image that is already in the registry, then the image is > > rebuilt > > (as this means something changed, very likely some dependency). But if the > > Rockerfile didn't change, then there is no need to rebuild the image, and > > just > > keep using the one already in the registry. This is also done for the LLVM > > images. This helps to improve the speed, as most of the times they don't > > need to > > be built again. > > It would be nice to have a GitLab CI variable which can be used to > force rebuilds of these regardless. > Oh, right! In our system, what we do is to use the scheduled pipelines to force this rebuild. The schedule is run once per week. I'll include this option then. > > The third stage is the one that builds Mesa itself. Here we just define > > which > > tool to use and which LLVM version. This is done by passing the right > > parameters > > to the `rocker build` tool. It will pick the right base image, install the > > missing dependencies (mainly, libdrm), select which drivers should be built > > (based on the LLVM version and parsing the configure.ac file), and create > > the > > image. > > You can eke out a little bit of a speed improvement by using the > go-faster runes from before 'apt-get update' here: > > https://gitlab.freedesktop.org/wayland/weston/blob/master/.gitlab-ci.yml#L6 > Cool! I think we can include it in the base image, so it gets propagated to rest of the builds. > Anyway, this all looks great, and I'm really excited to see it! The series is: > Acked-by: Daniel Stone <dani...@collabora.com> > > ... but please do not deploy it just yet. Right now we only have a > single CI runner for all of fd.o, which is a pretty elderly (SNB?) > Xeon, on a different continent from our GitLab instance. The network > is chokingly bad, but even if it weren't we'd be bottlenecked on > machine time. I'm working around to try to find sponsorship to get us > some more runners, located in decent data centres in the US, and will > let you know as soon as anything happens there. In the meantime, if > you have any ideas or know any people who would be interested in > sponsoring runners (very ideally US-based) or just covering the bill > for time on cloud VMs, please let me know! > Sure. Probably there are other improvements I can do meanwhile. I'll keep unpushed until everything ready and you grant green light. > Cheers, > Daniel > _______________________________________________ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
