I think this remark in the referenced link is the best summary of "what could
be improved":
"""the biggest weakness of the sstate signature bits, in my opinion, is that
it only tracks inputs, not outputs. If task A depends on B, and the metadata
input to B changes, then A will be rebuilt, even if the *output* of B didn't
change as a result of the change to its metadata."""
For example, if someone fixes a bug in gcc that only applies to MIPS, then
builds that only target an ARM machine shouldn't be affected. Much worse than
that, I have a dependency like:
gcc -> ... -> python -> bitstream tool -> FPGA image
so this means that a change in gcc causes Python to rebuild, which causes the
bitstream tool to be rebuilt and produce idential output, and that triggers a
roughly 31-hour build of lots of FPGA images. These are the ones we really
want to break. A binary output compare would help a lot, even if 80% of the
libraries fail to create reproducible output, I may still be spared those 31
hours...
I think tracking digital output is technically feasible, and won't require any
change to existing recipes.
Also think about "feed" setups. When I see my settop reporting "331 packages
need updating"... It'd be great if that could be avoided.
On 01-07-17 09:48, Martin Jansa wrote:
I haven't tried the patches, but I really like this idea (I was suggesting
something like that since 2011
http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327) and
I'm glad you weren't discouraged attempting to do this.
It also implements 3) b) idea from
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be
interested read that ticket.
Thanks
On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho <michael...@bmw-carit.de
<mailto:michael...@bmw-carit.de>> wrote:
Hi, at BMW Car IT we are working on an experimental feature to improve
sstate
cache hits and we are looking for comments on the approach who might have
some
insights to the problem and seeing if anyone is interested in the feature
for
mainline.
The sstate-cache of a recipe is tied closely to its build dependencies, as
the
signature generated for a task includes all parent task's signatures as
part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.
What this patchset does is propose to add another optional variable to
recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for
signature
generation it excludes any parent tasks that come from dependencies listed
in
it. This is to break the signature change domino effect.
This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks
that use
CDEPENDS and allow for deciding to re-execute a task despite being covered
by
sstate-cache.
The patchset is based on the jethro branch for the poky repository, as this
is
the branch that we are using. If the general idea sounds good, we can
consider
porting it to master.
Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility
report
containing the file checksums of all files that it packages and which file
paths
they are at. Another recipe, cdepends-test2, can then strip this
compatibility
report to the minimal files it needs to be consistent and can compare the
latest
checksums it used to configure/compile/install with and if they have
changed,
trigger a rebuild. If not, the previous version restored from sstate-cache
is
used.
We are still experimenting with the usages of this, including the use of
having
abi-compliance-checker to compare the ABI of shared libraries as a
compatibility
checker during RunQueue and using the results to avoid rebuilding child
recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change
a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.
We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
didn't meet
the feature requirements of what this compatibility checker callback is
doing,
although maybe when porting to master we could refactor to make better use
of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and
if
people are interested in making use of it and their use cases.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071 <tel:%2B49%20731%203780%204071>
Mobil: +49 152 5498 0471 <tel:%2B49%20152%205498%200471>
Fax: +49-731-37804-001 <tel:%2B49-731-37804-001>
Mail: michael...@bmw-carit.de <mailto:michael...@bmw-carit.de>
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
--
Kind regards,
Mike Looijmans
System Expert
TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijm...@topicproducts.com
Website: www.topicproducts.com
Please consider the environment before printing this e-mail
Visit us at the Space Tech Expo Europe (Stand E-71)
_______________________________________________
yocto mailing list
yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto
<https://lists.yoctoproject.org/listinfo/yocto>
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
