Hi Darcy, Thank you for the feedback. Actually, depending on what exactly you mean by user supplied script, this patch set for the concept already supports that.
I'm not sure if my attachments got skewered by the mailing list or not but if you can see my patch "0009-recipes-cdepends-test-example-recipe-framework-for-u.patch", you can see an example implementation and what needs to be done to introduce a custom compatibility checker. This patch set provides the hooks for Bitbake and recipes to communicate about the compatible dependency checker it needs to use. In essence, the recipes point Bitbake to a python module that it can load from any meta-layer lib folder and run a predefined function, passing the directories of the compatibility reports (the latest from the freshly built recipe, and the older from the sstate cache of the child recipe) which are also defined by the recipes. This python function can then do whatever it wants with the compatibility data it has access to (i.e. call a custom user script to look at the report information) and return "incompatible" or "compatible" to control the RunQueue. Bitbake will rebuild if incompatible is returned, and not rebuild if compatible is returned. It can also return a different value which Bitbake will interpret as a compatibility error (e.g. binary delivery recipe that will be incompatible so you want to crash the build instead of detecting the error at runtime). Therefore, the recipes themselves define all the compatibility report information and how to do the comparison, allowing for "user supplied hook scripts" which can be defined at any meta-layer level. In the case you're hinting towards, you would need to implement the following which can already hook into these patches: 1. a report generation function for the kernel recipe to output the data you need to compare - i.e. a compatibility report (a directory of files) that consists of the file checksums of the kernel headers stored in a text file and the PV of the recipe stored in another text file - to do so the kernel recipe merely needs to inherit compatible-depends-report and add a function that outputs these report files and add this function to the prefuncs of the task do_compatibility_report. 2. A python function to compare the different versions of the report format suggested in (1). 3. Modifying the child recipes to add the kernel to CDEPENDS and point Bitbake to the python function by setting the variable CDEPENDS_TOOL_xxx for the kernel provider. If you look at the patch 0009, you'll see the example is fairly simple but similar to what I explained above. The python function added to the lib folder is the actual compatibility comparison function and more can be easily introduced in any meta-layer to match the compatibility checking situation. (If interested, I can share another example compatibility checker that uses the PV of a recipe to do rebuild decisions). Happy to answer in more detail if you require, just let me know. Thank you. Kind regards, Michael Ho -- BMW Car IT GmbH Michael Ho Spezialist Entwicklung - Linux Software Integration Lise-Meitner-Str. 14 89081 Ulm Tel.: +49 731 3780 4071 Mobil: +49 152 5498 0471 Fax: +49-731-37804-001 Mail: michael...@bmw-carit.de Web: http://www.bmw-carit.de -------------------------------------------------------- BMW Car IT GmbH Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin Sitz und Registergericht: München HRB 134810 -------------------------------------------------------- From: Darcy Watkins <dwatk...@sierrawireless.com> Sent: 30 June 2017 17:01 To: Michael Ho Cc: yocto@yoctoproject.org Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache Hi, It would also be nice to add some sort of hook script support where user supplied hook script performs a check and returns either a "yes it's OK" or a "no have to rebuild it" status. And then tie that into the dependency logic. I have unusual use case where I need to have a dependency on changes to the kernel version, but not necessarily something triggered after every kernel rebuild. The package I'm building will trigger OpenSSL and a whole whack of stuff each time it rebuilds. It depends on the kernel version and the headers, but shouldn't need to depend on every patch or kconfig change that happens. ___ Regards, Darcy Darcy Watkins :: Senior Staff Engineer, Firmware SIERRA WIRELESS Direct +1 604 233 7989 :: Fax +1 604 231 1109 :: Main +1 604 231 1100 13811 Wireless Way :: Richmond, BC Canada V6V 3A4 [M4] dwatk...@sierrawireless.com :: www.sierrawireless.com :: www.inmotiontechnology.com On Jun 30, 2017, at 2:53 AM, Michael Ho <michael...@bmw-carit.de> wrote: Hi, at BMW Car IT we are working on an experimental feature to improve sstate cache hits and we are looking for comments on the approach who might have some insights to the problem and seeing if anyone is interested in the feature for mainline. The sstate-cache of a recipe is tied closely to its build dependencies, as the signature generated for a task includes all parent task's signatures as part of the signature information. This means that when changes occur in the parent recipes, even if insignificant, all children recipes that have valid sstate cache must invalidate their sstate cache objects. What this patchset does is propose to add another optional variable to recipes, CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature generation it excludes any parent tasks that come from dependencies listed in it. This is to break the signature change domino effect. This patchset also proposes modifying RunQueue to then be able to run a compatibility checker during task execution phase for recipes and tasks that use CDEPENDS and allow for deciding to re-execute a task despite being covered by sstate-cache. The patchset is based on the jethro branch for the poky repository, as this is the branch that we are using. If the general idea sounds good, we can consider porting it to master. Included is an patch that adds an example recipe and compatibility checker, where compatibility is based on the file checksums of the parent recipes packages. An example recipe, cdepends-test1, generates a compatibility report containing the file checksums of all files that it packages and which file paths they are at. Another recipe, cdepends-test2, can then strip this compatibility report to the minimal files it needs to be consistent and can compare the latest checksums it used to configure/compile/install with and if they have changed, trigger a rebuild. If not, the previous version restored from sstate-cache is used. We are still experimenting with the usages of this, including the use of having abi-compliance-checker to compare the ABI of shared libraries as a compatibility checker during RunQueue and using the results to avoid rebuilding child recipes when the .so files they depend on are still compatible. Example use cases of this are allowing recipes which provide multiple shared libraries to change a single .so file without rebuilding all its children that depend on the other shared libraries but not the one that changed. We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet the feature requirements of what this compatibility checker callback is doing, although maybe when porting to master we could refactor to make better use of the work already done there. The current implementation is a bit hacky but comments would be appreciated in regards to if the concept is feasible and if people are interested in making use of it and their use cases. Kind regards, Michael Ho -- BMW Car IT GmbH Michael Ho Spezialist Entwicklung - Linux Software Integration Lise-Meitner-Str. 14 89081 Ulm Tel.: +49 731 3780 4071 Mobil: +49 152 5498 0471 Fax: +49-731-37804-001 Mail: michael...@bmw-carit.de Web: http://www.bmw-carit.de -------------------------------------------------------- BMW Car IT GmbH Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin Sitz und Registergericht: München HRB 134810 -------------------------------------------------------- <0001-cache.py-add-support-for-CDEPENDS.patch> <0002-siggen.py-add-support-for-CDEPENDS.patch> <0003-runqueue.py-add-support-for-CDEPENDS.patch> <0004-taskdata.py-add-support-for-CDEPENDS.patch> <0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch> <0006-native.bbclass-add-support-for-CDEPENDS.patch> <0007-nativesdk.bbclass-add-support-for-CDEPENDS.patch> <0008-classes-add-bbclasses-compatible-depends-and-compati.patch> <0009-recipes-cdepends-test-example-recipe-framework-for-u.patch> -- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto -- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
