On Sun, Mar 10, 2024 at 11:52 AM Richard Purdie
<richard.pur...@linuxfoundation.org> wrote:
>
> On Sun, 2024-03-10 at 06:20 -0700, Richard Purdie via
> lists.openembedded.org wrote:
> > On Sun, 2024-03-10 at 13:31 +0100, Max wrote:
> > > Am Samstag, dem 09.03.2024 um 13:04 -0800 schrieb Bruce Ashfield:
> > > > On Sat, Mar 9, 2024 at 12:58 PM <max.oss...@gmail.com> wrote:
> > > > >
> > > > > From: Max Krummenacher <max.krummenac...@toradex.com>
> > > > >
> > > > > Hello
> > > > >
> > > > > If one builds a kernel using AUTOREV invoking bitbake only
> > > > > works
> > > > > once.
> > > > > Any subsequent bitbake invocation fails parsing the meta data.
> > > > >
> > > > > Reproducable with:
> > > > > - latest poky, b5624ee564
> > > > > - Kernel with SRCREV = "AUTOREV", e.g. in local.conf
> > > > > `SRCREV_machine:pn-linux-yocto:forcevariable = "${AUTOREV}"`
> > > > > - bitbake virtual/kernel; bitbake virtual/kernel
> > > > >
> > > > > On the second invocation parsing fails when the fetcher tries
> > > > > to
> > > > > evaluate the latest SRCREV:
> > > > >
> > > > > > ERROR: ExpansionError during parsing meta/recipes-
> > > > > > kernel/linux/linux-yocto_6.6.bb
> > > > > > Traceback (most recent call last):
> > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1245, in
> > > > > > srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > d.setVar("__BBAUTOREV_ACTED_UPON", True)
> > > > > > > srcrev = ud.method.latest_revision(ud, d, name)
> > > > > >
> > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 1667, in
> > > > > > Git.latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, name='machine'):
> > > > > > except KeyError:
> > > > > > > revs[key] = rev = self._latest_revision(ud,
> > > > > > d,
> > > > > > name)
> > > > > > return rev
> > > > > > File "bitbake/lib/bb/fetch2/git.py", line 850, in
> > > > > > Git._latest_revision(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, name='machine'):
> > > > > >
> > > > > > > output = self._lsremote(ud, d, "")
> > > > > > # Tags of the form ^{} may not work, need to
> > > > > > fallback to other form
> > > > > > File "bitbake/lib/bb/fetch2/git.py", line 833, in
> > > > > > Git._lsremote(ud=<bb.fetch2.FetchData object at
> > > > > > 0x7f8e26f5f290>, d=<bb.data_smart.DataSmart object at
> > > > > > 0x7f8e26195890>, search=''):
> > > > > > bb.fetch2.check_network_access(d, cmd,
> > > > > > repourl)
> > > > > > > output = runfetchcmd(cmd, d, True)
> > > > > > if not output:
> > > > > > File "bitbake/lib/bb/fetch2/__init__.py", line 957, in
> > > > > > runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
> > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > export
> > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > linux/usr/bin:scripts:build/tmp/work/qemux86_64-poky-
> > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > core.pager=cat
> > > > > > -c safe.bareRepository=all ls-remote
> > > > > > https://git.yoctoproject.org/linux-yocto.git ',
> > > > > > d=<bb.data_smart.DataSmart object at 0x7f8e26195890>,
> > > > > > quiet=True, cleanup=[], log=None, workdir=None):
> > > > > >
> > > > > > > raise FetchError(error_message)
> > > > > >
> > > > > > bb.data_smart.ExpansionError: Failure expanding variable
> > > > > > fetcher_hashes_dummyfunc[vardepvalue], expression was
> > > > > > ${@bb.fetch.get_hashvalue(d)} which triggered exception
> > > > > > FetchError: Fetcher failure: Fetch command export
> > > > > > PSEUDO_DISABLED=1; export
> > > > > > DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1003/bus";
> > > > > > export
> > > > > > PATH="build/tmp/sysroots-uninative/x86_64-
> > > > > > linux/usr/bin:scripts:/var/home/krm/build/poky/build/tmp/work
> > > > > > /q
> > > > > > emux86_64-poky-linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin/x86_64-poky-linux:build/tmp/work/qemux86_64-
> > > > > > poky-linux/linux-yocto/6.6.20+git/recipe-
> > > > > > sysroot/usr/bin/crossscripts:build/tmp/work/qemux86_64-poky-
> > > > > > linux/linux-yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/usr/bin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/sbin:build/tmp/work/qemux86_64-poky-linux/linux-
> > > > > > yocto/6.6.20+git/recipe-sysroot-
> > > > > > native/bin:bitbake/bin:build/tmp/hosttools"; export
> > > > > > HOME="/home/krm"; git -c gc.autoDetach=false -c
> > > > > > core.pager=cat
> > > > > > -c safe.bareRepository=all ls-remote
> > > > > > https://git.yoctoproject.org/linux-yocto.git failed with
> > > > > > exit
> > > > > > code 128, output:
> > > > > > fatal: unable to access
> > > > > > 'https://git.yoctoproject.org/linux-yocto.git/': error
> > > > > > setting
> > > > > > certificate file: build/tmp/work/x86_64-linux/curl-
> > > > > > native/8.6.0/recipe-sysroot-native/etc/ssl/certs/ca-
> > > > > > certificates.crt
> > > > > >
> > > > > > The variable dependency chain for the failure is:
> > > > > > fetcher_hashes_dummyfunc[vardepvalue]
> > > > >
> > > > > Note:
> > > > > One gets out of that parser error by deleting the git binary in
> > > > > the
> > > > > kernel's work recipe-sysroot-native
> > > > > `rm tmp/work/qemux86_64-poky-linux/linux-
> > > > > yocto/6.6.20+git/recipe-
> > > > > sysroot-native/usr/bin/git`
> > > > >
> > > > > Bisecting poky leads to commit
> > > > > f7fa98cca8 ("kern-tools: depend on git-replacement-native")
> > > > > Reverting it on top of b5624ee564 makes the parsing pass.
> > > > >
> > > > > I assume that `git-replacement-native` does not work with
> > > > > https,
> > > > > the
> > > > > fetch error also goes away if changing in SRC_URI from https to
> > > > > git.
> > > > >
> > > > > Any comments?
> > > >
> > > > I didn't even know that curl was coming into play :)
> > > >
> > > > Adding DEPENDS:class-native += "ca-certificates" to the curl
> > > > recipe
> > > > should resolve the issue.
> > >
> > > Looks like curl-native resp. libcurl hardcodes the lookup to its
> > > own
> > > work directory, i.e.:
> > > x86_64-linux/curl-native/8.6.0/recipe-sysroot-
> > > native/etc/ssl/certs/ca-certificates.crt
> > >
> > > So even if DEPENDS/RDEPENDS will install ca-certificates in the
> > > kernel's
> > > recipe-sysroot-native the parsing will fail if the curl-native
> > > directory
> > > is not/no longer populated, e.g. because curl-native came from
> > > sstate
> > > or
> > > rm_work is in INHERIT.
> >
> > This all gets a bit messy.
> >
> > We've relied upon scripts that use openssl to set variables like:
> >
> > export SSL_CERT_DIR="XXXX/etc/ssl/certs/
> >
> > so in theory we might be able to set an environment variable in a
> > wrapper around the git commands.
> >
> > It may be better if we teach curl a relative path to the certs...
> >
> > I suspect this isn't going to be an easy/neat fix unfortunately.
>
> I'm trying not to get sucked into "work" today however I realised that
> relative paths won't work without some implementation of "$ORIGIN"
> support into these paths. Given it ultimately ends up in openssl, it
> would probably be best there.
>
> I'd not be against writing a $ORIGIN support patch and seeing what
> upstream think about it. It would still mean finding a way to find the
> path to the library file somehow.
>
> For purposes of the release, setting the right envvars in the git
> wrapper is probably the way forward for now, much as I dislike the
> requirement to do that.
I tried this, but curl didn't seem to use it to locate the cert file.
i.e., I tried this:
---------------
diff --git a/meta/recipes-devtools/git/git_2.44.0.bb
b/meta/recipes-devtools/git/git_2.44.0.bb
index e6d1470873..f6b06ec601 100644
--- a/meta/recipes-devtools/git/git_2.44.0.bb
+++ b/meta/recipes-devtools/git/git_2.44.0.bb
@@ -31,7 +31,7 @@ PACKAGECONFIG ??= "expat curl"
PACKAGECONFIG[cvsserver] = ""
PACKAGECONFIG[svn] = ""
PACKAGECONFIG[manpages] = ",,asciidoc-native xmlto-native"
-PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl"
+PACKAGECONFIG[curl] = "--with-curl,--without-curl,curl ca-certificates"
PACKAGECONFIG[expat] = "--with-expat,--without-expat,expat"
EXTRA_OECONF = "--with-perl=${STAGING_BINDIR_NATIVE}/perl-native/perl \
@@ -103,13 +103,15 @@ do_install:append:class-target () {
do_install:append:class-native() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
- GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ GIT_TEMPLATE_DIR='`dirname
$''realpath`'/${REL_GIT_TEMPLATE_DIR} \
+ SSL_CERT_DIR='`dirname $''realpath`'/../../etc/ssl/certs/
}
do_install:append:class-nativesdk() {
create_wrapper ${D}${bindir}/git \
GIT_EXEC_PATH='`dirname $''realpath`'/${REL_GIT_EXEC_PATH} \
GIT_TEMPLATE_DIR='`dirname $''realpath`'/${REL_GIT_TEMPLATE_DIR}
+ SSL_CERT_DIR='`dirname $''realpath`'/../../etc/ssl/certs/
perl_native_fixup
}
---------
Probably because the exec of curl from git isn't getting the
environment ? Either
that, or I did it wrong. I didn't try it as an explicit export, and
that is probably it,
I can try that later tonight.
Only when I added the depends on ca-certifications to curl itself was it able to
fetch the autorevs.
Bruce
>
> This does feel like a problem it would be good to solve for wider Linux
> in general for relocatable (and reproducible) binaries.
>
> Cheers,
>
> Richard
>
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196899):
https://lists.openembedded.org/g/openembedded-core/message/196899
Mute This Topic: https://lists.openembedded.org/mt/104831542/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
