Ya, I need to look this over closely to try to understand your solution :) On Thu, Nov 21, 2024 at 10:17 AM Jia, Hongxu <hongxu....@windriver.com> wrote: > > Ping Joshua > > //Hongxu > > ________________________________ > From: openembedded-core@lists.openembedded.org > <openembedded-core@lists.openembedded.org> on behalf of hongxu via > lists.openembedded.org <hongxu.jia=eng.windriver....@lists.openembedded.org> > Sent: Wednesday, November 20, 2024 1:50 PM > To: openembedded-core@lists.openembedded.org > <openembedded-core@lists.openembedded.org>; jpewhac...@gmail.com > <jpewhac...@gmail.com>; Jia, Hongxu <hongxu....@windriver.com> > Subject: [OE-core] [RR V2][NULL][0/3] SPDX 3.0: Reduce redundant spdxid-hash > symlinks to save inode on host > > Summary: SPDX 3.0: Reduce redundant spdxid-hash symlinks to save inode on host > > * Git logs > [oe-core] > commit 5ebf6f058259bc073e83bd2fa2b3b116e6156d60 > Author: Hongxu Jia <hongxu....@windriver.com> > Date: Sat Nov 9 17:16:31 2024 +0800 > > oeqa/selftest: Add SPDX 3.0 include source cases for core_image_minimal > build > > $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source > 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: > 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib > 2024-11-20 03:29:24,850 - oe-selftest - INFO - > path-to/poky/meta-yocto-bsp/lib > 2024-11-20 03:29:24,850 - oe-selftest - INFO - > path-to/poky/meta-selftest/lib > 2024-11-20 03:29:24,850 - oe-selftest - INFO - > path-to/meta-openembedded/meta-oe/lib > 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base > configuration is valid/parsable > 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include > selftest.inc" in path-to/build_spdx3-st/conf/local.conf > 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include > bblayers.inc" in bblayers.conf > 2024-11-20 03:29:27,317 - oe-selftest - INFO - > test_core_image_minimal_include_source > (spdx.SPDX30Check.test_core_image_minimal_include_source) > 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message > 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok > 2024-11-20 05:21:12,798 - oe-selftest - INFO - > ---------------------------------------------------------------------- > 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s > 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK > 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: > 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - > spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) > 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: > 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 > test in 6706.272s > 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All > required tests passed (successes=1, skipped=0, failures=0, errors=0) > > Signed-off-by: Hongxu Jia <hongxu....@windriver.com> > > commit 3753b084cffcbef374093a442d4e692821d1a29b > Author: Hongxu Jia <hongxu....@windriver.com> > Date: Sat Nov 9 14:18:26 2024 +0800 > > sbom30.py: reduce redundant spdxid symlinks to save inode on host > > In order to support all in-scope SPDX data within a single > JSON-LD file for SPDX 3.0.1, Yocto's SBOM: > - In native/target/nativesdk recipe, created spdxid-hash symlink > for each element to point to the JSON-LD file that contains > element details; > - In image recipe, use spdxid-hash symlink to collect element > details from varies of JSON-LD files > > While SPDX_INCLUDE_SOURCES = "1", it adds sources to JSON-LD file > and create 2N+ spdxid-hash symlinks for N source files. > (N for software_File, N for hasDeclaredLicense's Relationship) > > For large numbers of source files, adding an extra symlink -> real file > will occupy one more inode (per file), which will need a slot in > the OS's inode cache. In this situation, disk performance is slow > and inode is used up quickly > > After commit [sbom30/spdx30: add link prefix and name to namespace > of spdxId and alias] applied, the namespace of spdxId and alias in > recipe and package jsonld differs. Use it to create symlink to jsonld, > take recipe shadow, package shadow and package shadow-src for example: > > For recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json > > spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/... > alias: recipe-shadow/UNIHASH/... > symlink: > tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/recipe-shadow.spdx.json -> > ../recipes/shadow.spdx.json > > For package jsonld > tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json > > spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/... > alias: package-shadow/UNIHASH/... > symlink: > tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow.spdx.json -> > ../packages/shadow.spdx.json > > In package jsonld > tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json > > spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/... > alias: package-shadow-src/UNIHASH/... > symlink: > tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-link/package-shadow-src.spdx.json -> > ../packages/shadow-src.spdx.json > > Build core-image-minimal with/without this commit, comparing the > spdxid-link > number, 7 281 824 -> 6 043 > > echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf > > Without this commit: > $ time bitbake core-image-minimal > real 100m17.769s > user 0m24.516s > sys 0m4.334s > > $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l > 7281824 > > With this commit: > $ time bitbake core-image-minimal > real 85m12.994s > user 0m20.423s > sys 0m4.228s > > $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l > 6043 > > Signed-off-by: Hongxu Jia <hongxu....@windriver.com> > > commit 0ea8bcaa9a8fe63f1935e02968fbaa541734024b > Author: Hongxu Jia <hongxu....@windriver.com> > Date: Tue Nov 19 20:12:37 2024 +0800 > > sbom30/spdx30: add link prefix and name to namespace of spdxId and alias > > In order to simple reference the SPDX ID to instead of making jsonld hash > path for each element, only creating one symlink for one file and > referencing > it multiple times, add link prefix and name to the namespace of spdxId > and alias > to replace ${PN} to avoid namespace conflict between recipe, packages and > images. > > Take recipe shadow, package shadow and package shadow-src for example: > Without this commit, spdxId and alias in recipe and package jsonld have > the same > namespace > > spdxId: http://spdx.org/spdxdocs/shadow-xxx/... > alias: shadow/UNIHASH/... > > After apply this commit, the namespace of spdxId in recipe and package > jsonld differs: > In recipe jsonld tmp/deploy/spdx/3.0.1/core2-64/recipes/shadow.spdx.json > > spdxId: http://spdx.org/spdxdocs/recipe-shadow-xxx/... > alias: recipe-shadow/UNIHASH/... > > In package jsonld tmp/deploy/spdx/3.0.1/core2-64/packages/shadow.spdx.json > > spdxId: http://spdx.org/spdxdocs/package-shadow-xxx/... > alias: package-shadow/UNIHASH/... > > In package jsonld > tmp/deploy/spdx/3.0.1/core2-64/packages/shadow-src.spdx.json > > spdxId: http://spdx.org/spdxdocs/package-shadow-src-xxx/... > alias: package-shadow-src/UNIHASH/... > > Then will use namespace of spdxId and alias to create link for jsonld > file, > one symlink for one jsonld file, referenced by elements multiple times > > Signed-off-by: Hongxu Jia <hongxu....@windriver.com> > > > > ====== Testing ====== > * Commands > > Build core-image-minimal with/without this commit, comparing the spdxid-link > number, 7 281 824 -> 6 043 > > echo 'SPDX_INCLUDE_SOURCES = "1"' >> local.conf > > Without this commit: > $ time bitbake core-image-minimal > real 100m17.769s > user 0m24.516s > sys 0m4.334s > > $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-hash -name "*.json" |wc -l > 7281824 > > With this commit: > $ time bitbake core-image-minimal > real 85m12.994s > user 0m20.423s > sys 0m4.228s > > $ find tmp/deploy/spdx/3.0.1/*/by-spdxid-link -name "*.json" |wc -l > 6043 > > > $ oe-selftest -r spdx.SPDX30Check.test_core_image_minimal_include_source > 2024-11-20 03:29:24,850 - oe-selftest - INFO - Adding layer libraries: > 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta/lib > 2024-11-20 03:29:24,850 - oe-selftest - INFO - > path-to/poky/meta-yocto-bsp/lib > 2024-11-20 03:29:24,850 - oe-selftest - INFO - path-to/poky/meta-selftest/lib > 2024-11-20 03:29:24,850 - oe-selftest - INFO - > path-to/meta-openembedded/meta-oe/lib > 2024-11-20 03:29:24,868 - oe-selftest - INFO - Checking base configuration is > valid/parsable > 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include selftest.inc" > in path-to/build_spdx3-st/conf/local.conf > 2024-11-20 03:29:27,317 - oe-selftest - INFO - Adding: "include bblayers.inc" > in bblayers.conf > 2024-11-20 03:29:27,317 - oe-selftest - INFO - > test_core_image_minimal_include_source > (spdx.SPDX30Check.test_core_image_minimal_include_source) > 2024-11-20 04:52:47,318 - oe-selftest - INFO - Keepalive message > 2024-11-20 05:14:08,115 - oe-selftest - INFO - ... ok > 2024-11-20 05:21:12,798 - oe-selftest - INFO - > ---------------------------------------------------------------------- > 2024-11-20 05:21:12,798 - oe-selftest - INFO - Ran 1 test in 6706.271s > 2024-11-20 05:21:12,798 - oe-selftest - INFO - OK > 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS: > 2024-11-20 05:21:32,026 - oe-selftest - INFO - RESULTS - > spdx.SPDX30Check.test_core_image_minimal_include_source: PASSED (6280.82s) > 2024-11-20 05:21:32,027 - oe-selftest - INFO - SUMMARY: > 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest () - Ran 1 test in > 6706.272s > 2024-11-20 05:21:32,027 - oe-selftest - INFO - oe-selftest - OK - All > required tests passed (successes=1, skipped=0, failures=0, errors=0) > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#207554): https://lists.openembedded.org/g/openembedded-core/message/207554 Mute This Topic: https://lists.openembedded.org/mt/109680228/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
