Ping Joshua
//Hongxu
________________________________
From: openembedded-core@lists.openembedded.org
<openembedded-core@lists.openembedded.org> on behalf of hongxu via
lists.openembedded.org <hongxu.jia=eng.windriver....@lists.openembedded.org>
Sent: Wednesday, November 20, 2024 1:50 PM
To: openembedded-core@lists.openembedded.org
<openembedded-core@lists.openembedded.org>; jpewhac...@gmail.com
<jpewhac...@gmail.com>; Jia, Hongxu <hongxu....@windriver.com>
Subject: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash
symlinks to save inode on host
Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host
* Git logs
[oe-core]
commit 5ebf6f058259bc073e83bd2fa2b3b116e6156d60
Author: Hongxu Jia <hongxu....@windriver.com>
Date: Sat Nov 9 17:16:31 2024 +0800
oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal
build
$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -
path-to/poky/meta-yocto-bsp/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -
path-to/poky/meta-selftest/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -
path-to/meta-openembedded/meta-oe/lib
2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration
is valid/parsable
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include
selftest.inc" in path-to/build_spdx3-st/conf/local.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include
bblayers.inc" in bblayers.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO -
test_core_image_minimal_include_source
(spdx.SPDX30Check.test_core_image_minimal_include_source)
2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok
2024-11-20 05:21:12,798 - oe-selftest - INFO -
----------------------------------------------------------------------
2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS -
spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test
in 6706.272s
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All
required tests passed (successes=1, skipped=0, failures=0, errors=0)
Signed-off-by: Hongxu Jia <hongxu....@windriver.com>
commit 3753b084cffcbef374093a442d4e692821d1a29b
Author: Hongxu Jia <hongxu....@windriver.com>
Date: Sat Nov 9 14:18:26 2024 +0800
sbom30.py: reduce redundant spdxid symlinks to save inode on host
In order to support all in-scope SPDX data within a single
JSON-LD file for SPDX 3.0.1, Yocto's SBOM:
- In native/target/nativesdk recipe, created spdxid-hash symlink
for each element to point to the JSON-LD file that contains
element details;
- In image recipe, use spdxid-hash symlink to collect element
details from varies of JSON-LD files
While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file
and create 2N+ spdxid-hash symlinks for N source files.
(N for software_File, N for hasDeclaredLicense's Relationship)
For large numbers of source files, adding an extra symlink -> real file
will occupy one more inode (per file), which will need a slot in
the OS's inode cache. In this situation, disk performance is slow
and inode is used up quickly
After commit [sbom30/spdx30: add link prefix and name to namespace
of spdxId and alias] applied, the namespace of spdxId and alias in
recipe and package jsonld differs. Use it to create symlink to jsonld,
take recipe shadow, package shadow and package shadow-src for example:
For recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
symlink:
tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json ->
../recipes/shadow.spdx.json
For package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
symlink:
tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json ->
../packages/shadow.spdx.json
In package jsonld
tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
symlink:
tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json ->
../packages/shadow-src.spdx.json
Build core-image-minimal with/without this commit, comparing the spdxid-link
number, 7 281 824 -> 6 043
echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf
Without this commit:
$ time bitbake core-image-minimal
real 100m17.769s
user 0m24.516s
sys 0m4.334s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l
7281824
With this commit:
$ time bitbake core-image-minimal
real 85m12.994s
user 0m20.423s
sys 0m4.228s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l
6043
Signed-off-by: Hongxu Jia <hongxu....@windriver.com>
commit 0ea8bcaa9a8fe63f1935e02968fbaa541734024b
Author: Hongxu Jia <hongxu....@windriver.com>
Date: Tue Nov 19 20:12:37 2024 +0800
sbom30/spdx30: add link prefix and name to namespace of spdxId and alias
In order to simple reference the SPDX ID to instead of making jsonld hash
path for each element, only creating one symlink for one file and
referencing
it multiple times, add link prefix and name to the namespace of spdxId and
alias
to replace ${PN} to avoid namespace conflict between recipe, packages and
images.
Take recipe shadow, package shadow and package shadow-src for example:
Without this commit, spdxId and alias in recipe and package jsonld have the
same
namespace
spdxId: http://spdx.org/spdxdocs/shadow-xxx/...
alias: shadow/UNIHASH/...
After apply this commit, the namespace of spdxId in recipe and package
jsonld differs:
In recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/...
alias: recipe-shadow/UNIHASH/...
In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/...
alias: package-shadow/UNIHASH/...
In package jsonld
tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json
spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/...
alias: package-shadow-src/UNIHASH/...
Then will use namespace of spdxId and alias to create link for jsonld file,
one symlink for one jsonld file, referenced by elements multiple times
Signed-off-by: Hongxu Jia <hongxu....@windriver.com>
====== Testing ======
* Commands
Build core-image-minimal with/without this commit, comparing the spdxid-link
number, 7 281 824 -> 6 043
echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf
Without this commit:
$ time bitbake core-image-minimal
real 100m17.769s
user 0m24.516s
sys 0m4.334s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l
7281824
With this commit:
$ time bitbake core-image-minimal
real 85m12.994s
user 0m20.423s
sys 0m4.228s
$ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l
6043
$ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source
2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries:
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-yocto-bsp/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib
2024-11-20 03:29:24,850 - oe-selftest - INFO -
path-to/meta-openembedded/meta-oe/lib
2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is
valid/parsable
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc"
in path-to/build_spdx3-st/conf/local.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc"
in bblayers.conf
2024-11-20 03:29:27,317 - oe-selftest - INFO -
test_core_image_minimal_include_source
(spdx.SPDX30Check.test_core_image_minimal_include_source)
2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message
2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok
2024-11-20 05:21:12,798 - oe-selftest - INFO -
----------------------------------------------------------------------
2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s
2024-11-20 05:21:12,798 - oe-selftest - INFO - OK
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS:
2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS -
spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s)
2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY:
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in
6706.272s
2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All required
tests passed (successes=1, skipped=0, failures=0, errors=0)
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#207553):
https://lists.openembedded.org/g/openembedded-core/message/207553
Mute This Topic: https://lists.openembedded.org/mt/109680228/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
