Hi Richard, hi Alex On Mon, 2023-10-30 at 14:07 +0000, Richard Purdie wrote: > On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote: > > On Thu, 14 Sept 2023 at 14:56, Richard Purdie > > <richard.pur...@linuxfoundation.org> wrote: > > > There are design elements to this work. We need to work out how > > > we can > > > make eSDK and "normal" builds more similar and less of an > > > overhead to > > > switch between one and the other. A "bblock all" command does > > > partly > > > get you to an eSDK effectively so part of this may be switching > > > eSDK to > > > use the new lock command. What other differences are there? What > > > other > > > differences are necessary or make sense for the use cases eSDK > > > was > > > designed for? How would you turn an existing build into an eSDK > > > like > > > one? Could you provide a copy of a local build to someone else > > > easily > > > using something like eSDK's tooling? What does the eSDK look like > > > at > > > the end of this. One section we don't have good answers to yet is > > > setup > > > and configuration although I know you've started on some of that. > > > > So I see the following differences between esdk and normal modes: > > > > 1. Environment and tooling availability. > > > > a) esdk sets a number of variables from its initialization script > > that > > aid with cross-compiling components directly (e.g. the core use > > case > > of SDKs). Normal mode doesn't do that, but recently added > > meta-ide-support will generate a similar initialization script that > > will set up the same environment from the normal mode. There are > > tests > > and documentation for it. > > In that case, this one is something we can document as how to make > the > functionality available in the normal build. > > > b) PATH. eSDK has a number of items in PATH that point to various > > locations inside tmp/sysroots/, collectively they provide the > > cross-toolchain. > > > > eSDK also puts a selection of yocto tools into path - wic, devtool > > but > > not bitbake: > > > > ============================ > > alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/ > > total 48 > > lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 devtool -> > > ../../../../layers/poky/scripts/devtool > > lrwxrwxrwx 1 alex alex 54 Oct 30 12:52 oe-find-native-sysroot -> > > ../../../../layers/poky/scripts/oe-find-native-sysroot > > lrwxrwxrwx 1 alex alex 42 Oct 30 12:52 recipetool -> > > ../../../../layers/poky/scripts/recipetool > > lrwxrwxrwx 1 alex alex 39 Oct 30 12:52 runqemu -> > > ../../../../layers/poky/scripts/runqemu > > lrwxrwxrwx 1 alex alex 55 Oct 30 12:52 runqemu-addptable2image -> > > ../../../../layers/poky/scripts/runqemu-addptable2image > > lrwxrwxrwx 1 alex alex 53 Oct 30 12:52 runqemu-export-rootfs -> > > ../../../../layers/poky/scripts/runqemu-export-rootfs > > lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-extract-sdk -> > > ../../../../layers/poky/scripts/runqemu-extract-sdk > > lrwxrwxrwx 1 alex alex 51 Oct 30 12:52 runqemu-gen-tapdevs -> > > ../../../../layers/poky/scripts/runqemu-gen-tapdevs > > lrwxrwxrwx 1 alex alex 46 Oct 30 12:52 runqemu-ifdown -> > > ../../../../layers/poky/scripts/runqemu-ifdown > > lrwxrwxrwx 1 alex alex 44 Oct 30 12:52 runqemu-ifup -> > > ../../../../layers/poky/scripts/runqemu-ifup > > lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd -> > > ../../../../tmp/work/qemuarm64-poky-linux/core-image- > > minimal/1.0/recipe-sysroot-native/usr/bin/unfsd > > lrwxrwxrwx 1 alex alex 35 Oct 30 12:52 wic -> > > ../../../../layers/poky/scripts/wic > > ============================== > > > > 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH. > > Cross-toolchain can be added by the same environment script made by > > meta-ide-support as mentioned in 1a. > > Right, so in theory we can change PATH and change this which can also > easily be documented. > > > 2. Configuration (e.g. local.conf).
I think these differences between SDK and bitbake environment are no longer required and they have been problematic. I would try to make the bitbake environment usable like the eSDK environment was without trying to replicate all the details of the eSDK installer such as these local.conf settings. > > > > eSDK local.conf is local.conf from the normal mode that was used to > > produce eSDK, stripped of all comments, and with a bunch of extra > > settings: > > > > ============================ > > INHERIT:remove = "buildhistory icecc" That's not needed it there is a full bitbake environment. > > CONNECTIVITY_CHECK_URIS = "" > > > > SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none" > > > > SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" As already mentioned in my previous mail, a locked SDK is not a must if the full bitbake environment with shared sstate-cache is available. Locking might be added later as an optional feature. > > > > BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES" > > > > BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir > > *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot" > > > > BUILDCFG_HEADER = "" > > > > METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" > > > > # Provide a flag to indicate we are in the EXT_SDK Context > > WITHIN_EXT_SDK = "1" > > > > SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1 > > file://universal-4.9/(.*) file://universal-4.8/\1" > > As already mentioned in my previous mail, the SSTATE_MIRRORS does not work well. Downloading the sstate-cache to SSTATE_DIR works much better, at least for us. > > Perhaps some of this should become a generic include file which is > included which would then make it easy to document as how to lock > things down to match it? That would take the config out of the class > file too which is probably nicer. > > > require conf/locked-sigs.inc > > require conf/unlocked-sigs.inc > > =============================== > > > > Devtool also has a special configuration: > > ================= > > alex@Zen2:~/poky_sdk$ cat conf/devtool.conf > > [General] > > bitbake_subdir = layers/poky/bitbake > > init_path = layers/poky/oe-init-build-env > > core_meta_subdir = layers/poky/meta > The different folder structures of the eSDK and the bitbake environment cause many issues in the past. We should get rid of that and keep the layers structure form the bitbake environment. But that's already the case if the layers are replicated with the bitbake-layers tools from Alex. > > These are likely there to allow devtool to work with the modified > files > layout. > > We should probably document it but I'm going to propose we "ignore" > layout for the purposes of this so we can just ignore things which > are > for layout only. > > > [SDK] > > sdk_targets = core-image-minimal > > Not quite sure about this one. Maybe something to do with the target > used to generate the eSDK? Not sure what uses it. In the eSDK you can do "devtool build" without specifying the image which should be used. That's probably then taken from this config line. But also this seams to be pointless in a full bitbake environment. > > > > ================== > > There is currently no tooling to add/remove these extras in either > > esdk mode or normal mode as far as I understand. Their individual > > purposes and effects are also not exactly clear to me, and need to > > be > > investigated one by one. > > > > 3. Setting up a normal mode in a eSDK installation. > > > > This is actually pretty easy: rather than sourcing the sdk > > environment > > script, source the poky/oe-init-build-env: > > ============== > > alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env . > > > > ### Shell environment set up for builds. ### > > ... > > =============== > > Bitbake will then simply run, albeit with all those extra > > esdk-specific items in local.conf (which can be easily taken out to > > achieve a pristine normal mode). There will be some warnings but > > builds will proceed: > > =============== > > WARNING: You are using a local hash equivalence server but have > > configured an sstate mirror. This will likely mean no sstate will > > match from the mirror. You may wish to disable the hash equivalence > > use (BB_HASHSERVE), or use a hash equivalence server alongside the > > sstate mirror. > > ... > > WARNING: spirv-headers-native-1_1.3.261.1-r0 do_fetch: Failed to > > fetch > > URL git://github.com/KhronosGroup/SPIRV- > > Headers;protocol=https;branch=main, > > attempting MIRRORS if available > > WARNING: libepoxy-1.5.10-r0 do_fetch: Failed to fetch URL > > git://github.com/anholt/libepoxy;branch=master;protocol=https, > > attempting MIRRORS if available > > WARNING: spirv-tools-native-1_1.3.261.1-r0 do_fetch: Failed to > > fetch > > URL git://github.com/KhronosGroup/SPIRV- > > Tools.git;branch=main;protocol=https, > > attempting MIRRORS if available > > WARNING: shaderc-native-2023.6-r0 do_fetch: Failed to fetch URL > > git://github.com/google/shaderc.git;protocol=https;branch=main, > > attempting MIRRORS if available > > =============== > > 4. Setting up esdk mode in a normal yocto installation. That's exactly what my devtool ide plugin does if it gets called without a recipe. See https://patchwork.yoctoproject.org/project/oe-core/patch/20231101110129.647878-9-adrian.freiho...@siemens.com/ and search for *Shared sysroots mode*. > > > > This is on one hand easy: cross-toolchain and unified sysroots are > > made available for direct component builds by meta-ide-support and > > build-sysroots. > > > > On the other hand, setting up the exact same environment as eSDK > > sets > > up isn't currently possible: tooling to replicate all the > > local.conf > > tweaks (that includes locked-sigs items) would be necessary, at a > > minimum. A more pedantic approach would also ensure that > > - only the 'allowed' set of tools is in PATH, and not everything > > from > > bitbake/bin/ and scripts/ > > - environment variables and toolchain sysroots match between what > > eSDK > > sets up, and what meta-ide-support sets up > > > > 5. In conclusion: to be honest, right now I'm not sure which > > direction > > should be taken to unify and simplify things. Hopefully the above > > sparks some ideas; they might pop into my head later as well :-) > > The biggest 'gap' appears to be the config to go along with the > locked > sigs. If we move that to an include, that probably gets us quite > close > to them being similar ignoring layout changes for the files and the > differences to the environment? > > Thinking out loud, I guess we could have a "filtered" scripts > directory > too as part of the normal build, then the eSDK switches PATH from one > to the other? Simply speaking, the new populate_sdk_ext implementation could look like this: Create a tar file which contains: - Output of bitbake-layers create-layers-setup - Output of bitbake-layers save-build-conf - An sstate download script which can download the artifacts from a mirror into SSTATE_DIR. - Optionally: The populated sstate-cache folder. Generates a script which: - includes the tar file as the existing installer - extracts the tar and runs all the scripts from the tar. This would allow to initially setup the SDK with an installer. It would also allow to download a newer installer and install it over and existing setup. It would basically just checkout different commits of of the layers and update the sstate-cache. But it would also allow to update the SDK independently from an installer, by just using git to change the commits of the layers and re-run the sstate-download script again. This architecture would also allow to switch for example the MACHINE or compile another image. And, please have a look at the devtool ide plugin. It was developed with all that in mind. Regards, Adrian > > Cheers, > > Richard > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190042): https://lists.openembedded.org/g/openembedded-core/message/190042 Mute This Topic: https://lists.openembedded.org/mt/102322302/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
