On Mon, 2026-07-06 at 15:21 +0100, Richard Purdie via lists.openembedded.org
wrote:
> On Sat, 2026-04-18 at 20:34 +0200, Adam Blank via lists.openembedded.org
> wrote:
> > It has lain there for a while, obscuring certain aspects of
> > how task signatures are calculated.
> >
> > This changeset does not change the current status quo, but
> > rather changes the way in which it is implemented. It can be
> > followed by changes which further refine what is excluded and
> > how, from those tasks which refer to 'extend_recipe_sysroot'.
> >
> > Related discussion:
> > https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.ca...@linuxfoundation.org/
> >
> > Signed-off-by: Adam Blank <[email protected]>
> > ---
> > Adam Blank (7):
> > package_pkgdata: fix typo to stop calling undefined function
> > staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
> > staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
> > cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
> > native: add 'extend_recipe_sysroot' to 'vardepsexclude'
> > wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude'
> > bitbake.conf: remove 'extend_recipe_sysroot' from
> > BB_HASHEXCLUDE_COMMON
> >
> > meta/classes-global/package_pkgdata.bbclass | 4 +++-
> > meta/classes-global/staging.bbclass | 5 ++++-
> > meta/classes-recipe/cross.bbclass | 1 +
> > meta/classes-recipe/native.bbclass | 1 +
> > meta/conf/bitbake.conf | 2 +-
> > meta/recipes-core/meta/wic-tools.bb | 1 +
> > 6 files changed, 11 insertions(+), 3 deletions(-)
>
> I finally found a bit of time to look at this. I took some of it, I
> ended up reworking some of the patches as they needed rewording to
> justify them and explain the changes being made.
>
> I didn't take the extend_recipe_sysroot removal from bitbake.conf as
> I'm still not convinced we have all the issues that might cause
> identified. The function name typo and the incorrect vardepsexclude are
> fixed though (or will be once a couple of patches pass testing/review
> and merge).
I still wasn't 100% happy with this so I did go a little further and
hacked up the attached script. It does have one hardcoded path but you
can get the idea of what I'm trying to do.
You can start with a clean TMPDIR and run "bitbake world -S none". You
can then edit bitbake.conf and remove extend_recipe_sysroot from
BB_HASHEXCLUDE_COMMON and run that command a second time.
If you them run the attached script in tmp/stamps, after a bit of
processing, you will see:
Added {'staging_processfixme[func]', 'staging_copydir', 'BB_RUNTASK',
'oe.utils.get_multilib_datastore', 'sstate_clean_manifest',
'extend_recipe_sysroot', 'DEFAULTTUNE', 'RECIPE_SYSROOT',
'oe.sstatesig.find_sstate_manifest', 'TUNE_PKGARCH:tune-x86-64-v3',
'STAGING_DIR', 'extend_recipe_sysroot[func]', 'RECIPE_SYSROOT_NATIVE',
'COMPONENTS_DIR', 'staging_copydir[func]', 'staging_copyfile[func]',
'sstate_clean_manifest[func]', 'staging_populate_sysroot_dir',
'setscene_depvalid', 'PN', 'oe.path.remove', 'TUNE_PKGARCH',
'RECIPE_SYSROOT_MANIFEST_SUBDIR', 'TARGET_ARCH',
'staging_populate_sysroot_dir[func]', 'staging_processfixme',
'setscene_depvalid[func]', 'staging_copyfile'}
Removed set()
That means those variables were added as dependencies somewhere in the
build for at least one task. This was basically my worry with the
patch. We need to work out where/why those are being added and triage
them in order to be able to remove extend_recipe_sysroot from the hash
exclusion. Some of them may be acceptable, some of them may not, a lot
depends on context.
I'm still not sure it is worth the work in changing that, but we do now
at least have a way of checking the effect of such a change.
Cheers,
Richard
#!/usr/bin/env python3
import sys
import os
import re
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
sys.path.insert(0, '/media/build/poky/bitbake/lib')
from bb import __version__ as libbb_version
__version__ = "2.19.0"
if __version__ != libbb_version:
sys.exit("Bitbake core library version and program version mismatch!")
import bb
import bb.compress.zstd
import json
class SetEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, set) or isinstance(obj, frozenset):
return dict(_set_object=list(sorted(obj)))
return json.JSONEncoder.default(self, obj)
def SetDecoder(dct):
if '_set_object' in dct:
return frozenset(dct['_set_object'])
return dct
def dict_diff(a, b, ignored_vars=set()):
sa = set(a.keys())
sb = set(b.keys())
common = sa & sb
changed = set()
for i in common:
if a[i] != b[i] and i not in ignored_vars:
changed.add(i)
added = sb - sa
removed = sa - sb
return changed, added, removed
def get_data(fn):
try:
with bb.compress.zstd.open(fn, "rt", encoding="utf-8", num_threads=1) as f:
a_data = json.load(f, object_hook=SetDecoder)
except (TypeError, OSError) as err:
bb.error("Failed to open sigdata file '%s': %s" % (fn, str(err)))
raise err
return a_data
pattern = re.compile(r'^[\d.]+\.(do_[a-z_]+)\.sigdata\..*')
tasksplit = {}
for root, dirs, files in os.walk("."):
for f in files:
if ".sigdata." not in f:
continue
m = pattern.match(f)
if not m:
continue
task = m.group(1) + "-" + root
if task not in tasksplit:
tasksplit[task] = []
tasksplit[task].append(root + "/" + f)
adds = set()
removes = set()
for task in tasksplit:
files = tasksplit[task]
if len(files) == 1:
continue
if len(files) != 2:
print("Invalid number of sigdata files %s" % files)
continue
files = sorted(files, key=lambda f: os.path.getmtime(f))
a_data = get_data(files[0])
b_data = get_data(files[1])
changed, added, removed = dict_diff(a_data['gendeps'], b_data['gendeps'], a_data['basehash_ignore_vars'] & b_data['basehash_ignore_vars'])
adds.update(added)
removes.update(removed)
print("Added %s" % adds)
print("Removed %s" % removes)
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#240379):
https://lists.openembedded.org/g/openembedded-core/message/240379
Mute This Topic: https://lists.openembedded.org/mt/118895154/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
