[OE-core][PATCH] spdx30: Improve os.walk() handling

2025-02-13 Thread Joshua Watt via lists.openembedded.org
There have been errors seen when assembling root file system SPDX documents where they will references files that don't exist in the package SPDX. The speculation is that this is caused by os.walk() ignoring errors when walking, causing files to be omitted. Improve the code by adding an error hand

Re: [OE-core][PATCH] classes/create-spdx-2.2: Fix SPDXDIR affecting multiconfig machines

2025-02-12 Thread Joshua Watt via lists.openembedded.org
Based on this patch, I'm guessing you're not working off of master? The patch doesn't apply to the latest master version because a lot of the variables in create-spdx-2.2 were moved to spdx-common.bbclass. I tried the reproduction steps on master and it seems to work OK, but I'll keep trying; can

[OE-core][PATCH] scripts/contrib: Add oe-image-files-spdx script

2025-02-11 Thread Joshua Watt via lists.openembedded.org
Adds a template for a python project that processes the SPDX 3.0.1 output from a build and lists all the files on the root file system with their checksums This is intended to be an example to show how to deal with the SPDX data to do common tasks. Signed-off-by: Joshua Watt --- .../contrib/oe-

Re: [OE-core][PATCH] classes/create-spdx-2.2: Fix SPDXDIR affecting multiconfig machines

2025-02-10 Thread Joshua Watt via lists.openembedded.org
Piotr, I think with the provided example I can look into this. I suspect it will be a little more involved than adding MACHINE_ARCH On Mon, Feb 10, 2025 at 9:58 AM Piotr Łobacz wrote: > > Actually this was just an issue with the path to the deps.json file. > which was containing all the dependen

Re: [OE-core][PATCH] bitbake.conf: Include fragments before local.conf

2025-02-05 Thread Joshua Watt via lists.openembedded.org
On Wed, Feb 5, 2025 at 3:37 PM Alexander Kanavin wrote: > > Btw the other option for overriding fragments would be to use direct require > statement with a path to them in local.conf, followed by tweaks. It’s not too > horrible. And using a fragment variable can then be seen as ‘immutable’ > fr

Re: [OE-core][PATCH] bitbake.conf: Include fragments before local.conf

2025-02-05 Thread Joshua Watt via lists.openembedded.org
On Wed, Feb 5, 2025 at 3:26 PM Alexander Kanavin wrote: > > One could conceivably want to ship a local.conf.sample template with > fragments already enabled in it, or hand edit local.conf to enable something > via fragment. We can’t force users to use fragment tooling. > > It would be equally su

Re: [OE-core][PATCH] bitbake.conf: Include fragments before local.conf

2025-02-05 Thread Joshua Watt via lists.openembedded.org
On Wed, Feb 5, 2025 at 2:36 PM Alexander Kanavin wrote: > > We have discussed this before. With this change, users will no longer be able > to set fragments from local.conf, as addfragments would be processed before > that file. You need to use ?= in fragments instead for things that are meant

[OE-core][PATCH] bitbake.conf: Include fragments before local.conf

2025-02-05 Thread Joshua Watt via lists.openembedded.org
Per some usability assessment and offline discussion, one of the useful use cases of config fragments is that they allow the "default" configuration for some device to be committed to source control and easily selected and used by the end users. However, when doing this, there is still the desire t

[OE-core][PATCH] spdx30: Include files in rootfs

2025-01-30 Thread Joshua Watt via lists.openembedded.org
Adds a "contains" relationship that relates the root file system package to the files contained in it. If a package provides a file with a matching hash and path, it will be linked, otherwise a new File element will be created Signed-off-by: Joshua Watt --- meta/lib/oe/spdx30_tasks.py | 46 +

[OE-core][PATCH] lib/oe/sbom30: Fix SHA256 hash dictionary

2025-01-30 Thread Joshua Watt via lists.openembedded.org
Fixes a bug in the code that created the hash dictionary that was accidentally excluding items with a SHA256 hash instead of including them Signed-off-by: Joshua Watt --- meta/lib/oe/sbom30.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe

Re: [OE-core] [PATCH V3 2/3] meta/lib/oe/spdx30_tasks.py: add patched CVE to SPDX 3

2025-01-29 Thread Joshua Watt via lists.openembedded.org
On Wed, Jan 8, 2025 at 9:05 AM Hongxu Jia wrote: > > Due to commit [cve-check: annotate CVEs during analysis][1] improved > get_patched_cves to search for additional CVEs from CVE_STATUS which > means the funciton get_patched_cves contains both of patched CVE and > decoded_status > > This commit a

Re: [OE-core] [RFC PATCH] lib/spdx30_tasks: support directories deployed by image recipes

2025-01-29 Thread Joshua Watt via lists.openembedded.org
LGTM Reviewed-by: Joshua Watt On Mon, Jan 13, 2025 at 5:34 AM Igor Opaniuk via lists.openembedded.org wrote: > > From: Igor Opaniuk > > create_image_spdx() implementation assumes that image is indeed a file. > If image recipe deploys a directory (for example, which contains an > hierarchy of f

Re: [OE-core] [PATCH] create-spdx: support line numbers

2025-01-21 Thread Joshua Watt via lists.openembedded.org
On Mon, Jan 20, 2025 at 6:10 AM Denis OSTERLAND-HEIM via lists.openembedded.org wrote: > > LIC_FILES_CHKSUM supports begin-/endline for licenses included in > for instance header files. This patch adds support for line numbers > to NO_GENERIC_LICENSE, too. > > Signed-off-by: Denis Osterland-Heim

[OE-core][PATCH] lib: spdx: Upgrade to final 3.0.1 release

2025-01-16 Thread Joshua Watt via lists.openembedded.org
The 3.0.1 release of SPDX has been officially released with a few minor modifications. Regenerate the bindings to use this version. Signed-off-by: Joshua Watt --- meta/lib/oe/sbom30.py | 2 +- meta/lib/oe/spdx30.py | 146 -- 2 files changed, 85 insertion

Re: [OE-core] [PATCH v2] lib/spdx30_tasks: support directories deployed by image recipes

2025-01-13 Thread Joshua Watt via lists.openembedded.org
On Mon, Jan 13, 2025 at 10:51 AM wrote: > > From: Igor Opaniuk > > create_image_spdx() implementation assumes that image is indeed a file. > If image recipe deploys a directory (for example, which contains an > hierarchy of flash artifacts, that is used by SoC vendor-specific > flashing tool) whi

Re: [OE-core] [RFC PATCH] lib/spdx30_tasks: support directories deployed by image recipes

2025-01-13 Thread Joshua Watt via lists.openembedded.org
On Mon, Jan 13, 2025 at 5:34 AM Igor Opaniuk via lists.openembedded.org wrote: > > From: Igor Opaniuk > > create_image_spdx() implementation assumes that image is indeed a file. > If image recipe deploys a directory (for example, which contains an > hierarchy of flash artifacts, that is used by S

[OE-core][PATCH] udev-extraconf: Allow FAT mount group to be specified

2025-01-10 Thread Joshua Watt via lists.openembedded.org
Adds a variable that allows the group given permissions to access FAT mounted file systems to be specified, instead of being hardcoded to 'disk'. As a note, the usage of 'disk' as the mount group is not a very secure default, since the disk user usually has read/write access to all block devices i

Re: [OE-core] [PATCH v4 2/2] meta/lib/oe/sbom30.py: fix alias in simplelicensing_customIdToUri not extracted

2025-01-07 Thread Joshua Watt via lists.openembedded.org
Reviewed-by: Joshua Watt On Tue, Dec 24, 2024 at 7:48 PM hongxu via lists.openembedded.org wrote: > > After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set > license_text_map with SPDX alias other than actual ID > > The property of simplelicensing_customIdToUri is > ListPr

Re: [OE-core] [PATCH 1/2] meta/lib/oe/spdx30_tasks.py: set license alias to hasConcludedLicense relationship

2025-01-07 Thread Joshua Watt via lists.openembedded.org
Reviewed-by: Joshua Watt On Wed, Dec 25, 2024 at 10:43 PM Hongxu Jia wrote: > > Set license alias other than spdxId to hasConcludedLicense relationship > > $ echo 'MACHINE = "qemux86-64"' >> conf/local.conf > $ bitbake shadow > $ In tmp/deploy/spdx/3.0.1/corei7-64/recipes/recipe-shadow.spdx.json

Re: [OE-core] [PATCH] meta/classes/create-spdx.bbclass: support to inherit SPDX class conditionally

2025-01-07 Thread Joshua Watt via lists.openembedded.org
Reviewed-by: Joshua Watt On Thu, Dec 26, 2024 at 11:11 PM Hongxu Jia wrote: > > In order to inherit SPDX class conditionally, using variable ${SPDX_CLASS} > to inherit SPDX class other than original 'INHERIT:remove = "create-spdx"' > and INHERIT += "create-spdx-2.2" > > Signed-off-by: Hongxu Jia

Re: [OE-core] [PATCH 2/2] meta/lib/oe/sbom30.py: set alias for scan declared licenses

2025-01-07 Thread Joshua Watt via lists.openembedded.org
Reviewed-by: Joshua Watt On Wed, Dec 25, 2024 at 10:43 PM Hongxu Jia wrote: > > when using SPDX_INCLUDE_SOURCES, it calls scan_declared_licenses > to scan licenses from source file, set alias for the newly added > license and and license alias to hasDeclaredLicense relationship > > $ echo 'MACHI

Re: [OE-core] [PATCH] meta/lib/oe/spdx30_tasks.py: add patched CVE to SPDX 3

2025-01-07 Thread Joshua Watt via lists.openembedded.org
On Mon, Dec 30, 2024 at 12:40 AM Hongxu Jia wrote: > > Some CVEs may be patched during the build process without incrementing > the version number, save the CVEs fixed by patches to cve_by_status > which was parsed as "Patched" status to add new_vex_patched_relationship > to pkg_objset > > Take re

Re: [OE-core] [PATCH v3 3/3] bitbake-config-build: add a plugin for config fragments

2024-12-27 Thread Joshua Watt via lists.openembedded.org
On Fri, Dec 27, 2024 at 11:44 AM Alexander Kanavin wrote: > > On Fri, 27 Dec 2024 at 19:12, Joshua Watt wrote: > > Ok, I tried it again, and it didn't work for me (or at least, didn't > > do what I would have liked). You can set MACHINE and DISTRO in config > > fragments, *but* it doesn't affect

Re: [OE-core] [PATCH v3 3/3] bitbake-config-build: add a plugin for config fragments

2024-12-27 Thread Joshua Watt via lists.openembedded.org
Ok, I tried it again, and it didn't work for me (or at least, didn't do what I would have liked). You can set MACHINE and DISTRO in config fragments, *but* it doesn't affect the machine or distro include in bitbake.conf. This means you can't really specify MACHINE or DISTRO in a fragment, or you're

Re: [OE-core] [PATCH v3 3/3] bitbake-config-build: add a plugin for config fragments

2024-12-25 Thread Joshua Watt via lists.openembedded.org
Hmm, ok, I'll try again. On Wed, Dec 25, 2024, 8:29 AM Alexander Kanavin wrote: > On Fri, 20 Dec 2024 at 18:39, Joshua Watt wrote: > > 1) bitbake-config-build is way too aggressive in the files it looks > > for as fragments. My editor (vim) creates hidden temporary files that > > I kept having

[OE-core][PATCH 2/2] lib: configfragements: enable/disable multiple fragements at once

2024-12-20 Thread Joshua Watt via lists.openembedded.org
Extends the 'enable-fragment' and 'disable-fragment' commands so that they accept multiple fragments at once as a convenience for the user Signed-off-by: Joshua Watt --- meta/lib/bbconfigbuild/configfragments.py | 31 +-- 1 file changed, 17 insertions(+), 14 deletions(-) dif

[OE-core][PATCH 1/2] lib: configfragments: Restrict fragment file checking

2024-12-20 Thread Joshua Watt via lists.openembedded.org
The current implementation of the config fragments is too aggressive in checking files; any file in the fragment directory is checked, including hidden files or files with weird extensions. In particular, if an editor is creating temporary backup files when editing, these will be checked and will a

Re: [OE-core] [PATCH v3 3/3] bitbake-config-build: add a plugin for config fragments

2024-12-20 Thread Joshua Watt via lists.openembedded.org
OK, I gave this a try and here is some feedback: 1) bitbake-config-build is way too aggressive in the files it looks for as fragments. My editor (vim) creates hidden temporary files that I kept having to manually delete or the tool would attempt to parse them and fail. I would recommend that it on

Re: [OE-core] [PATCH V2] meta/lib/oe/sbom30.py: fix alias in simplelicensing_customIdToUri not extracted

2024-12-16 Thread Joshua Watt via lists.openembedded.org
On Mon, Dec 16, 2024 at 1:06 AM Hongxu Jia wrote: > > After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set > license_text_map with SPDX alias other than actual ID > > The property of simplelicensing_customIdToUri is > ListProp(ObjectProp(DictionaryEntry))), > and class Dict

Re: [OE-core] [PATCH V2] meta/lib/oe/spdx30.py: make StringProp to decode SPDX alias with actual ID

2024-12-16 Thread Joshua Watt via lists.openembedded.org
On Sun, Dec 15, 2024 at 10:20 PM Hongxu Jia wrote: > > After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set > license_text_map with SPDX alias other than actual ID > > The property of simplelicensing_customIdToUri is > ListProp(ObjectProp(DictionaryEntry))), > and class Dic

Re: [OE-core] [PATCH] meta/lib/oe/spdx30_tasks.py: fix alias in simplelicensing_customIdToUri not extracted

2024-12-13 Thread Joshua Watt via lists.openembedded.org
On Thu, Dec 12, 2024 at 4:09 AM Hongxu Jia wrote: > > After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set > license_text_map with alias other than actual spdxId > > The property of simplelicensing_customIdToUri is > ListProp(ObjectProp(DictionaryEntry))), > and class Dicti

Re: [OE-core] [PATCH 3/3] gcc-cross.inc: add var-SSTATE_ARCHS_TUNEPKG to vardeps of do_create_spdx

2024-12-11 Thread Joshua Watt via lists.openembedded.org
On Wed, Dec 11, 2024 at 5:07 AM Hongxu Jia wrote: > > The recipe gcc-cross inherited cross.bbclass which had class-native > recipe behavior, but depends on class-target recipe linux-libc-headers. > It caused the tweaking of ${TUNE_PKGARCH} for 32bit arm did not trigger > rebuild of do_create_spdx

[OE-core][PATCH] spdx 3.0: Rework how SPDX aliases are linked

2024-12-10 Thread Joshua Watt via lists.openembedded.org
The SPDX code needs to be able to look up an Element by its SPDX ID, locating the file that (should) contain the SPDX ID and opening it for parsing. Previously, the code would do this be hashing each Element SPDX ID and Alias, and the creating a symbolic link to the file that contains the element w

[OE-core][PATCH] classes-global/insane: Handle case where RPROVIDER is also a provider

2024-12-10 Thread Joshua Watt via lists.openembedded.org
The check to see if a provider of a given package is listed first checks for an exact match of the provider name. However, if this match existed, but didn't match in the task dependencies, it would not continue to look for other providers of package. This would manifest if one (non-virtual) recipe

Re: [oe-core][PATCH 3/3] oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal build

2024-12-04 Thread Joshua Watt via lists.openembedded.org
Is there some advantage to this patch over the test_gcc_include_source test? On Mon, Nov 25, 2024 at 1:15 AM Hongxu Jia wrote: > > $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source > 2024-11-09 09:17:54,600 - oe-selftest - INFO - Adding layer libraries: > 2024-11-09 09:17:5

[OE-core][PATCH 1/5] lib: sbom30: Fix import handling

2024-12-03 Thread Joshua Watt via lists.openembedded.org
Fixes a bug in the way that imports where being handled, due to a misspelled variable Signed-off-by: Joshua Watt --- meta/lib/oe/sbom30.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 0a7b4c05fbe..ea70f6fb69d 100644 --- a

[OE-core][PATCH 5/5] lib: oeqa: spdx: Add tests for extra options

2024-12-03 Thread Joshua Watt via lists.openembedded.org
Adds a test for several of the extra options provided by the SPDX classes. In particular, these are the options that can produce non-reproducible results, so are not enabled by default in OE core. This test takes care to configure the build so that the tests do run in a reproducible manner so that

[OE-core][PATCH 4/5] lib: spdx: Fix SPDX_BUILD_HOST

2024-12-03 Thread Joshua Watt via lists.openembedded.org
The call to make a new import for the SPDX_BUILD_HOST was incorrect, since it was asking for the key named "SPDX_BUILD_HOST", instead of the key name contained in that variable Signed-off-by: Joshua Watt --- meta/lib/oe/spdx30_tasks.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff

[OE-core][PATCH 2/5] lib: sbom30: Fix agent reference

2024-12-03 Thread Joshua Watt via lists.openembedded.org
When a agent reference was being used, the code was not using the correct base variable to look up the agent Signed-off-by: Joshua Watt --- meta/lib/oe/sbom30.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index ea70f6fb69d..7

[OE-core][PATCH 3/5] classes: create-spdx: Fix variable dependencies

2024-12-03 Thread Joshua Watt via lists.openembedded.org
A few variable dependencies were incorrect from the SPDX tasks, which prevented rebuilds from occurring, or caused them to occur when they should not. Signed-off-by: Joshua Watt --- meta/classes-recipe/create-spdx-image-3.0.bbclass | 3 +++ meta/classes/create-spdx-3.0.bbclass | 10

[OE-core][PATCH 0/5] Various SPDX Fixes

2024-12-03 Thread Joshua Watt via lists.openembedded.org
In the course of making training material for SPDX, I discovered that several of SPDX options to add additional information about the build were broken. Fix these, and add a selftest to make sure that they continue working in the future. Joshua Watt (5): lib: sbom30: Fix import handling lib: s

Re: [oe-core][PATCH V3 1/3] sbom30/spdx30: add link prefix to the namespace of spdxId and alias

2024-12-02 Thread Joshua Watt via lists.openembedded.org
OK, thanks. I did get some time to sit down, and read through this, and I understand the problem that needs to be solved. I think you're on the right track, but I think we can use the rework to remove some code that is no longer necessary. My basic thought is now: 1. Set an OEIdAliasExtension on t

[OE-core][PATCH] lib: spdx30_tasks: Fix supplied By

2024-12-02 Thread Joshua Watt via lists.openembedded.org
Fixes the supplied by property in SPDX to use the correct name Signed-off-by: Joshua Watt --- meta/lib/oe/spdx30_tasks.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 5aeed5cd6fc..30f1a3a1661 100644 --- a/meta

[OE-core][PATCH] lib: oe: license: Add missing import

2024-11-26 Thread Joshua Watt via lists.openembedded.org
The code in this file uses oe.qa, but it was not imported resulting an an exception when a license error was detected Signed-off-by: Joshua Watt --- meta/lib/oe/license.py | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py index 32c77fa204d..6f882

[OE-core][PATCH] classes/go-vendor: Use UNPACKDIR instead of WORKDIR

2024-11-26 Thread Joshua Watt via lists.openembedded.org
modules.txt is provided using file:// in SRC_URI, therefore it will be found in UNPACKDIR instead of WORKDIR Signed-off-by: Joshua Watt --- meta/classes/go-vendor.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.b

[OE-core][PATCH] spdx: Add do_deploy_build_sbom_spdx

2024-11-22 Thread Joshua Watt via lists.openembedded.org
Adds a task that can be used to generate a stand alone SBoM for a given build of a recipe. This SBoM includes the inputs and output for the build, as well as the build and runtime dependencies. The dependencies however will be added as unresolved imports to the OE SPDX alias, so as to avoid pulling

[OE-core][PATCH] lib: sbom30: Prefix aliases with "http://spdx.org/spdxdocs/"

2024-11-22 Thread Joshua Watt via lists.openembedded.org
While the SPDX ID aliases are primarily intended for internal use by the SPDX code, they are used in places where a valid SPDX ID is expected. In order to make sure that they still conform to this, prefix them with "http://spdx.org/spdxdocs/openembedded-alias"; Signed-off-by: Joshua Watt --- met

Re: [OE-core][PATCH] spdx: Fix SPDX tasks not running when code changes

2024-11-22 Thread Joshua Watt via lists.openembedded.org
On Fri, Nov 22, 2024 at 9:22 AM Quentin Schulz wrote: > > Hi Joshua, > > On 11/22/24 4:49 PM, Joshua Watt via lists.openembedded.org wrote: > > The SPDX code makes heavy use of python classes. While this works very > > well, the bitbake dependency parser is unable t

[OE-core][PATCH] spdx: Fix SPDX tasks not running when code changes

2024-11-22 Thread Joshua Watt via lists.openembedded.org
The SPDX code makes heavy use of python classes. While this works very well, the bitbake dependency parser is unable to understand how to deal with them, and thus changes to the class code do not cause rebuilds to occur. To correct this, add the library files that include SPDX code as file checksum

[OE-core][PATCH] lib: sbom30: Fix build parameters

2024-11-22 Thread Joshua Watt via lists.openembedded.org
Fixes the build parameters code, since the build_parameter property was changed to be singular in SPDX 3.0.1 Signed-off-by: Joshua Watt --- meta/lib/oe/sbom30.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index e3a9428668e..8

Re: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host

2024-11-21 Thread Joshua Watt via lists.openembedded.org
Ya, I need to look this over closely to try to understand your solution :) On Thu, Nov 21, 2024 at 10:17 AM Jia, Hongxu wrote: > > Ping Joshua > > //Hongxu > > > From: openembedded-core@lists.openembedded.org > on behalf of hongxu via > lists.openembedded.org

Re: [OE-core] [PATCH 1/3] sbom30.py: reduce redundant spdxid-hash symlinks to save inode on host

2024-11-18 Thread Joshua Watt via lists.openembedded.org
I think what you are trying to get at here is that the same file is present in multiple different "packages", so all references are equally interchangeable? I'm fine if we can figure out a reasonable way to do that, but I don't think this is the correct approach. A better option would be to simple