Hello,
I propose that CI keeps the ccache of each Linux build job between runs.
A run on master fills the caches, and each PR build restores them. Below
are the proposal, the measurements, and a forecast of the savings.
1. Today
--------
cibuild.sh -c already enables ccache in the Linux jobs. But the cache
directory is in the job's workspace and no step saves it. Each job
starts with an empty cache, and ccache only helps when the same file is
compiled again in the same job. apache/nuttx uses 0 bytes of its GitHub
Actions cache today.
Numbers from the GitHub API, 2026-09-26 to 2026-10-08 (13 full days,
340 completed build runs, almost all pull_request):
- All build workflow jobs: 581 runner-hours per day.
- Linux build jobs: 538 runner-hours per day.
- One run, 29 Linux jobs: 33 runner-hours (run 37919588548).
- "Run builds" step: 92 % of the Linux job time. The rest
is Docker pull, checkout and set-up.
- ccache size per job at the
end of the job: 0.07 GB to 0.87 GB, 11.6 GB in total.
- ccache hits inside one job: 22 % to 80 %.
2. Proposal
-----------
a) A new workflow runs the Linux build matrix on master and saves the
ccache directory of each job (key ccache-<job>-<run id>). It runs:
- every night, and
- on a push to master that changes a core header (the headers that
most objects include: include/*.h, include/sys/*.h, the common
include/nuttx headers, arch/*/include). A concurrency group with
cancel-in-progress makes a burst of merges start only one run.
b) The PR build jobs restore the newest cache of the same job
(actions/cache/restore, restore-keys ccache-<job>-). PR jobs do not
save caches. A PR can read the caches of its base branch, also from
a fork.
c) GitHub gives a repository 10 GB of cache. The 29 caches need 11.6 GB
today, so ccache max_size must be about 300 MB per job (8.7 GB in
total). The 16 jobs that now use more than 300 MB will keep only a
part of their cache. Another possibility is to ask Infra for a larger
cache.
3. Measurements
---------------
I ran the arm-03 test list (44 configurations, lpc17xx_40xx) on my
machine the way CI runs it (testbuild.sh -j 8 -e "-Wno-cpp -Werror",
ccache in front of the compilers). The cold build has the same ccache
statistics as the CI job (38.6 % hits and 0.2 GB, CI: 39 % and 0.20 GB).
Cache Time ccache hits
--------------------------------------------- ------- -----------
Empty (what CI does today) 1412 s 38.7 %
From the night before, no core header change 1076 s 96.8 %
From the night before, core headers changed 1427 s 42.4 %
The cache saves 24 % of the build time. The rest of the time is
configure, link and clean of each configuration, which ccache does not
change. On 2026-10-09 master changed stdbool.h, string.h, unistd.h,
time.h and nuttx/sched.h, and a cache from the night before saved
nothing. That is why the proposal also starts a run when a core header
changes.
How often does that occur? In the last 90 days, 27 of the 1476 commits
on master changed a core header, on 19 of 90 days (21 %). With merges
less than 3 hours apart counted once, that is 0.21 extra runs per day.
4. Forecast
-----------
Run builds time of the Linux jobs: 538 h x 92 % = 495 h per day
Saving when the cache is valid: 495 h x 24 % = 119 h per day
Cache valid, nightly run only: about 90 % of the time
Cache valid, nightly + header runs: about 97 % of the time
A core header change makes the cache invalid until the next run on
master ends: on average about 12 hours with the nightly run only, and
about 3 hours (one run) with the header runs. 21 % of the days have
such a change.
Nightly only Nightly + header
Saving 106 h/day 116 h/day
Cost of the nightly run -33 h/day -33 h/day
Cost of the header runs -7 h/day
Net 73 h/day 76 h/day
The net saving is about 75 runner-hours per day, 13 % of all build
workflow time. The longest jobs also finish sooner, which shortens the
time until a PR has a result. The header runs add little in total, but
they keep PR builds fast on the days when the core headers change.
Limits of the forecast: the 24 % comes from one test list on one machine
with 8 parallel jobs. GitHub runners have 4 CPUs, so compilation can be
a larger part of the time there, and the saving larger. Test lists with
larger configurations (sim, esp32) compile more per configuration. The
forecast also ignores the 300 MB limit of point 2c, which makes the
saving of the 16 large jobs smaller. I propose to run it for two weeks
and then compare the runner-hours per day with the numbers above.
5. Next steps
-------------
If there is no objection, I will open a PR with the new workflow and the
restore step in build.yml. The macOS and msys2 jobs can follow later:
msys2 uses 38 runner-hours per day and does not enable ccache today.
Best regards,
Marco Casaroli
