Hello,
I'm having a few problems when using the MIRRORS variable for GIT repos. I
see the same problems with the PREMIRRORS variable.
My use-case is:
1. company A has a private layer with SRC_URI that points to private git
2. company A needs to share layers with company B
3. company B cannot access company A's private git tree
4. company B has a mirror of the git trees with the same contents,
including SRC_URIs pointing to the company A private git trees
I have tried to resolve this using the MIRRORS variable in local.conf.
If SRCREV is a SHA, it seems to work for me.
However, it fails when the SRCREV is not a fixed SHA, eg. if SRCREV is a
branch, tag, or ${AUTOREV}. It looks like do_fetch is trying to pull the
repo to work out the SHA, but before decoding MIRRORS or PREMIRRORS. It
appears to do an ls-remote on the original repo, not the mirror, and fails.
The next issue is when using MIRRORS to replace a GIT repo with one using
SSH. This may simply be a syntax problem on my side.
Example:
MIRRORS += \
git://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git \
ssh://
g...@git.linaro.org/landing-teams/working/qualcomm/wcnss-config.git;protocol=ssh
\n \
"
The MIRROR doesn't seem to be used at all in the above example. I've tried
adding ";protocol=https" to the first line; it doesn't help, and fails in a
different way, giving a "Exception: KeyError: 'protocol'" error.
I've shown detailed examples below. People may not need them, but if my
email isn't clear enough, hopefully they will help explain.
Setup
--------------------------------------------------
For an example, I'm using Linaro's OE-RPB master. More info here:
https://github.com/96boards/oe-rpb-manifest
First, set up the environment:
$ mkdir -p ${WORKSPACE}
$ repo init -u https://github.com/96boards/oe-rpb-manifest.git -b master
$ repo sync
$ export MACHINE=dragonboard-410c ; export DISTRO=rpb ; .
setup-environment
Then, make sure it's working:
$ bitbake -c do_fetch wcnss-config
Mod a recipe to use a fake SRC_URI, and test that it no longer works.
$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#git.linaro.org#git.fake.org#g'
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config
I see this error output, as expected:
----------------------------------------------------
WARNING: wcnss-config-1.13-r0 do_fetch: Failed to fetch URL git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https,
attempting MIRRORS if available
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; LANG=C git -c core.fsyncobjectfiles=0 clone
--bare --mirror
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git
--progress failed with exit code 128, output:
Cloning into bare repository
'/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git'...
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure for URL: 'git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https'.
Unable to fetch URL from any source.
ERROR: wcnss-config-1.13-r0 do_fetch:
ERROR: wcnss-config-1.13-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in:
/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/temp/log.do_fetch.2525
ERROR: Task
(/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/wcnss-config_1.13.bb:do_fetch)
failed with exit code '1'
----------------------------------------------------
SRCREV PROBLEM
--------------------------------------------------
Set MIRRORS in local.conf and re-test. It should work again.
$ cat << EOF >> conf/local.conf
MIRRORS += " \\
git://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git \\
git://git.linaro.org/landing-teams/working/qualcomm/wcnss-config.git \n \\
"
EOF
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config
All I see in the build is a warning, and the downloads/git2 directory is
populated as expected:
$ ll
total 12K
lrwxrwxrwx 1 ryan ryan 110 Feb 20 17:15
git.fake.org.landing-teams.working.qualcomm.wcnss-config.git ->
/linaro/oe-rpb-workspace-testing/downloads/git2/git.linaro.org.landing-teams.working.qualcomm.wcnss-config.git/
-rw-rw-r-- 1 ryan ryan 6 Feb 20 17:15
git.fake.org.landing-teams.working.qualcomm.wcnss-config.git.done
drwxrwxr-x 7 ryan ryan 4.0K Feb 20 17:15
git.linaro.org.landing-teams.working.qualcomm.wcnss-config.git/
Change SRCREV to ${AUTOREV} and retest.
$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#^SRCREV.*#SRCREV = "${AUTOREV}"#'
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config
I see this failure:
ERROR: ExpansionError during parsing
/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb######################################
| ETA: 0:00:05
Traceback (most recent call last):
File
"/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/__init__.py", line
1170, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
name='default'):
if srcrev == "AUTOINC":
> srcrev = ud.method.latest_revision(ud, d, name)
File
"/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/__init__.py", line
1565, in Git.latest_revision(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
name='default'):
except KeyError:
> revs[key] = rev = self._latest_revision(ud, d, name)
return rev
File "/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/git.py",
line 594, in Git._latest_revision(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
name='default'):
"""
> output = self._lsremote(ud, d, "")
# Tags of the form ^{} may not work, need to fallback to other
form
File "/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/git.py",
line 583, in Git._lsremote(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
search=''):
bb.fetch2.check_network_access(d, cmd, repourl)
> output = runfetchcmd(cmd, d, True)
if not output:
File
"/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/__init__.py", line
888, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; git -c core.fsyncobjectfiles=0 ls-remote
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git ',
d=<bb.data_smart.DataSmart object at 0x7fd78988e748>, quiet=True,
cleanup=[], log=None, workdir=None):
> raise FetchError(error_message)
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression
was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; git -c core.fsyncobjectfiles=0 ls-remote
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
failed with exit code 128, output:
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org
A similar error happens if you use a tag rather than ${AUTOREV}:
$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#^SRCREV.*#SRCREV = "debian/1.13"#'
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config
These are the messages I see:
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; git -c core.fsyncobjectfiles=0 ls-remote
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
failed with exit code 128, output:
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org
ERROR: wcnss-config-1.13-r0 do_fetch:
ERROR: wcnss-config-1.13-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in:
/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/temp/log.do_fetch.5673
ERROR: Task
(/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/wcnss-config_1.13.bb:do_fetch)
failed with exit code '1'
SSH MIRRORS Problem
--------------------------------------------
>From a clean setup as above with the SRC_URI point to to git.fake.org...
$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#git.linaro.org#git.fake.org#g'
$ cat << EOF >> conf/local.conf
MIRRORS += " \\
git://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git \\
ssh://
g...@git.linaro.org/landing-teams/working/qualcomm/wcnss-config.git;protocol=ssh
\n \\
"
EOF
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config
These are the errors I see:
WARNING: wcnss-config-1.13-r0 do_fetch: Failed to fetch URL git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https,
attempting MIRRORS if available
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; LANG=C git -c core.fsyncobjectfiles=0 clone
--bare --mirror
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git
--progress failed with exit code 128, output:
Cloning into bare repository
'/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git'...
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure for URL: 'git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https'.
Unable to fetch URL from any source.
ERROR: wcnss-config-1.13-r0 do_fetch:
ERROR: wcnss-config-1.13-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in:
/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/temp/log.do_fetch.7701
ERROR: Task
(/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/wcnss-config_1.13.bb:do_fetch)
failed with exit code '1'
It looks like the MIRRORS variable hasn't replaced the repo at all.
Regards,
Ryan.
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
