All,
I've been working on making OE builds reproducible (that is, two given
builds can have binary-identical outputs). The current "test" for
reproducibility involves building core-image-minimal in two different
build directories, then doing a binary diff of the resulting target
Debian packages files and reporting if any of them differ (I'd like to
expand this test, see below). I believe that we are very close to
achieving this level of reproducibility, with a few caveats as shown below:
1. Both builds must be clean builds from scratch
2. Neither build can use sstate (sstate isn't currently reproducible for
a variety of reasons, more on that later)
3. The QA test for reproducibility takes about 4 hours on my 4/8 core
i7-3770 CPU @ 3.40GHz. I'm not sure how "expensive" a test has to be
before it can't reasonably be run on the autobuilders, but I'm guessing
this isn't a QA test that would currently be able to be run very often
(if at all). If sstate were reproducible, this would effectively be cut
in half, since you would only need one clean build from scratch (if that
would even matter).
The current test is obviously deficient in a few areas, but I believe
that is at the very least a good starting point since it has already
uncovered numerous reproducibility issues. The places where I think it
needs to be improved are:
1. Testing RPM and IPK package formats. I think RPMs will be pretty
easy; IPKs might be more challenging since AFAIK the tools that make
them don't generate reproducible output to begin with.
2. Testing more images than core-image-minimal; This should be pretty
straight forward to add to the QA test, it's mostly a matter of fixing
all the issues that come up.
3. Test for binary reproducible images (e.g. check that the entire ext4
image produced is binary identical). This one also might be pretty easy
for some formats, and hard for others (e.g. ext4 I think would be easy,
squashfs might be hard).
4. Improve the test to better test timestamp changes. Currently, the QA
test runs the two test builds serially which ensures that they have a
different datestamp when building. However, there are some packages that
are not reproducible based on only the Day, Month, or Year, neither of
which is likely to be different between the two serial test builds. I
would like to figure out a way to force one of the builds to be
separated by a sufficient about of time to tease out these issues. This
might be as easy as running bitbake under faketime, or it might be more
involved.
5. I don't know if anyone is clamoring for reproducible nativesdk builds?
6. We should also be testing if sstate objects are reproducible,
otherwise sstate can't really be relied on when doing a reproducible
build (In fact, I think the original reproducible build work that I took
over was focused on making sstate reproducible).
I think that OE has some significant advantages in being able to make
reproducible builds compared to other projects attempting the same
thing; primarily, we are capable of building up all (or most) of the
required build tools internally, then using these internal tools to
build up the target (e.g. we build GCC for the target, then use this
built GCC to compile target source). This means that we have a great
opportunity to isolate the build from the host environment and truly
achieve "simple" reproducible builds; any given set of layers at their
respective SHA's should be able to build a binary identical output on
any given host, with (ideally) no dependency on the host. We can't do
this today, and I've identified a number of roadblocks that will need to
be resolved (this is not a complete list; there will be more):
1. HOSTTOOLS differences. There are a lot of tools listed in HOSTTOOLS,
and unfortunately some of them have version dependent output and are
used for target builds (the one I've currently stumbled upon is pod2man,
but I'm sure there are others). Unfortunately, one could probably argue
that HOSTTOOLS is somewhat antithetical to the above statement, at least
in regard to target builds. Any host tool output that "leaks" into the
target build output can result in a non-reproducible build across hosts,
and possibly should be avoided; the alternative is to use (or mandate)
the corresponding -native recipe that provides that tool as a DEPENDS so
that the controlled internally built version is used instead. Note that
this only really applies target builds, not -native (or nativesdk right
now). -native recipes would obviously need more HOSTTOOLS to help
bootstrap the system. I suspect this would require reworking how
HOSTOOLS works so that they can be split into two categories somehow;
the tools that have "ubiquitous and stable" interfaces and are fine for
all recipes (e.g. cat, sed, true, rm, etc.) and those that are variable
and should only be used for -native builds (e.g. pod2man, rpcgen(?),
chrpath(?), tar(?)... others?). Anyone have thoughts on this?
2. sstate currently isn't reproducible. This is at least partially
related to the why non-clean rebuilds aren't reproducible[1]. These two
are related because AFAIK there isn't really anyway of knowing if an
sstate object came from a clean build of a recipe or a rebuild of a
recipe, so as long as rebuilds aren't reproducible, neither will sstate
be reproducible. The simplest fixes for these problems is to add more
-native tools to DEPENDS if they are used by the builds so that are
"stable" across all the tasks where it matters, but there might also be
some more "tricky" things that can/should be done with RSS to help
mitigate the problem. The HOSTTOOLS issue also makes sstate
non-reproducible, since AFAIK, there isn't necessarily a way to ensure
that a sstate object came from a specific host. In fact, I would
speculate that most core reproducibility issues will also make sstate
non-reproducible. Reproducible sstate also plays directly into hash
equivalence, since it is based on sstate and would be *much* more
effective if sstate were reproducible.
Many of the remaining problems can be solved by adding more -native
recipes to DEPENDS, but this has meet with some (justified) push back;
doing this things will likely increase the build time since more -native
dependencies will mean more -native tools have to be built, and more
serialization of the builds waiting for those tools to be built. I
suspect this is more true for replacing HOSTTOOLS with -native recipes,
since many of them may not have needed to be built at all. For the
sstate/rebuild reproducibility this is likely to have less impact since
those recipes were going to eventually have been built to be included in
RSS. Adding them to DEPENDS just moves them to be included sooner.
I'm curious what people thing about all this; How important is
reproducibility? How reproducible do we want to be? How hard should it
be to have reproducible builds? What trade-offs are willing to be made
for reproducible builds? Are there smart ways we can mitigate some of
the potential performance impacts of reproducible builds?
Thanks for your time. I know this was a long e-mail.
Joshua Watt
[1]: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13378
--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
