Hi , Just to note I have emailed DockerHub, asking for clarification on our account and what our benefits are.
On Thu, Oct 29, 2020 at 6:34 PM Allen Wittenauer <a...@effectivemachines.com.invalid> wrote: > > > On Oct 29, 2020, at 9:21 AM, Joan Touzet <woh...@apache.org> wrote: > > > > (Sidebar about the script's details) > > Sure. > > > I tried to read the shell script, but I'm not in the headspace to fully > parse it at the moment. If I'm understanding correctly, this will still > catch CouchDB's CI docker images if they haven't changed in a week, which > happens often enough, negating the cache. > > Correct. We actually tried something similar for a while and > discovered that in a lot of cases, upstream packages would disappear (or > worse, have security problems) thus making it look the image is still > "good" when it's not. So a rebuild weekly at least guarantees some level > of "yup, still good" without having too much of a negative impact. > > > As a project, we're kind of stuck between a rock and a hard place. We > want to force a docker pull on the base CI image if it's out of date or the > image is corrupted. Otherwise we want to cache forever, not just for a > week. I can probably manage the "do we need to re-pull?" bit with some > clever CI scripting (check for the latest image hash locally, validate the > local image, pull if either fails) but I don't understand how the script > resolves the latter. > > Most projects that use Yetus for their actual CI testing build the > image used for the CI as part of the CI. It is a multi-stage, multi-file > docker build that has each run use a 'base' Dockerfile (provided by the > project) that rarely changed and a per-run file that Yetus generates on the > fly, with both images tagged by either git sha or branch (depending upon > context). Due to how docker image reference counts on the layers work, this > makes the docker images effectively used as a "rolling cache" and (beyond a > potential weekly cache removal) full builds are rare.. thus making them > relatively cheap (typically <1m runtime) unless the base image had a change > far up the chain (so structure wisely). Of course, this also tests the > actual image of the CI build as part of the CI. (What tests the testers? > philosophy) Given that Jenkins tries really hard to have job affinity, > re-runs were still cheap after the initial one. [Ofc, now that the cache is > getting nuked every day....] > > Actually, looking at some of the ci-hadoop jobs, it looks like > yetus is managing the cache on them. I'm seeing individual run containers > from days ago at least. So that's a good sign. > > > Can a exemption list be passed to the script so that images matching a > certain regex are excluded? You say the script ignores labels entirely, so > perhaps not... > > Patches accepted. ;) > > FWIW, I've been testing on my local machine for unrelated reasons > and I keep blowing away running containers I care about so I might end up > adding it myself. That said: the code was specifically built for CI > systems where the expectation should be that nothing is permanent. > > -- *Gavin McDonald* Systems Administrator ASF Infrastructure Team
