Re: [yocto] yocto Digest, Vol 93, Issue 45
Hi Richard,
Did you manage to take a look at this patch ?
Regards,
Aaron
-Original Message-
From: yocto-boun...@yoctoproject.org [mailto:yocto-boun...@yoctoproject.org] On
Behalf Of yocto-requ...@yoctoproject.org
Sent: Wednesday, June 13, 2018 6:23 PM
To: yocto@yoctoproject.org
Subject: yocto Digest, Vol 93, Issue 45
Send yocto mailing list submissions to
yocto@yoctoproject.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.yoctoproject.org/listinfo/yocto
or, via email, send a message with subject or body 'help' to
yocto-requ...@yoctoproject.org
You can reach the person managing the list at
yocto-ow...@yoctoproject.org
When replying, please edit your Subject line so it is more specific than "Re:
Contents of yocto digest..."
Today's Topics:
1. [PATCH] [yocto-autobuilder2] Add support to enable Manual BSP
on LAVA (Aaron Chan)
2. Re: mono-native is trying to install files into a shared
area... (Alex Lennon)
3. Re: Image specific configuration files (Iv?n Castell)
--
Message: 1
Date: Wed, 13 Jun 2018 15:07:58 +0800
From: Aaron Chan
To: yocto@yoctoproject.org
Cc: Aaron Chan
Subject: [yocto] [PATCH] [yocto-autobuilder2] Add support to enable
Manual BSP on LAVA
Message-ID:
<1528873678-17502-1-git-send-email-aaron.chun.yew.c...@intel.com>
Signed-off-by: Aaron Chan
---
config.py | 9 +
schedulers.py | 9 +++--
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/config.py b/config.py
index 2568768..d21948f 100644
--- a/config.py
+++ b/config.py
@@ -80,3 +80,12 @@ builder_to_workers = {
"nightly-deb-non-deb": [],
"default": workers
}
+
+# Supported LAVA-Linaro on Yocto Project # Enable Automated
+Manual(Hardware) BSP Test case(s) enable_hw_test = {
+"enable": False,
+"lava_user" : "",
+"lava_token" : "",
+"lava_server" : ":"
+}
diff --git a/schedulers.py b/schedulers.py index 8f3dbc5..2c1b8e1 100644
--- a/schedulers.py
+++ b/schedulers.py
@@ -63,9 +63,14 @@ def props_for_builder(builder):
props.append(util.BooleanParameter(
name="deploy_artifacts",
label="Do we want to deploy artifacts? ",
-default=Boolean
+default=False
+))
+if builder in ['nightly-x86-64', 'nightly-x86-64-lsb', 'nightly-arm',
'nightly-arm-lsb', 'nightly-arm64']:
+props.append(util.BooleanParameter(
+name="enable_hw_test",
+label="Enable BSP Test case(s) on Hardware?",
+default=config.enable_hw_test['enable']
))
-
props = props + repos_for_builder(builder)
return props
--
2.7.4
--
Message: 2
Date: Wed, 13 Jun 2018 10:50:49 +0100
From: Alex Lennon
To: Craig McQueen
Cc: "yocto@yoctoproject.org"
Subject: Re: [yocto] mono-native is trying to install files into a
shared area...
Message-ID:
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 12/06/2018 05:43, Khem Raj wrote:
>
>
> On Mon, Jun 11, 2018 at 8:36 PM Craig McQueen
> mailto:craig.mcqu...@innerrange.com>>
> wrote:
>
> I wrote:
> >
> > I wrote:
> > >
> > > Lately, I'm trying to upgrade to a later version of mono, 5.4.1.6.
> > > When I try to do a build of my Yocto image, bitbake gets to
> the end of
> > > building mono- native, and then gets an error:
> > >
> > >
> > > ERROR: mono-native-5.4.1.6-r0 do_populate_sysroot: The recipe
> mono-
> > > native is trying to install files into a shared area when
> those files already
> > exist.
> > > Those files and their manifest location are:
> > > /home/craigm/yocto/poky/build/tmp/sysroots/x86_64-
> > > linux/usr/lib/mono/lldb/mono.py
> > >? Matched in b''
> > > /home/craigm/yocto/poky/build/tmp/sysroots/x86_64-
> > > linux/usr/lib/mono/4.6.1-api/System.Web.Http.SelfHost.dll
> > >? Matched in b''
> > > ...
> > > /home/craigm/yocto/poky/build/tmp/sysroots/x86_64-
> > >
> >
> linux/usr/lib/mono/xbuild/14.0/bin/MSBuild/Microsoft.Build.CommonTypes.
> > > xsd
> > >? Matched in b''
> > > /home/craigm/yocto/poky/build/tmp/sysroots/x86_64-
> > >
> linux/usr/lib/mono/xbuild/14.0/bin/MSBuild/Microsoft.Build.Core.xsd
> > >? Matched in b''
> > > /home/craigm/yocto/poky/build/tmp/sysroots/x86_64-
> > >
> >
> linux/usr/lib/mono/xbuild/14.0/Microsoft.Common.targets/ImportAfter/Mi
> > > c
> > > rosoft.NuGet.ImportAfter.targets
> > >? Matched in b''
> > > Please verify which recipe should provide the above files.
> > > The build has stopped as continuing in this scenario WILL break
> > > things, if not now, possibly in the future (we've seen builds fail
> > > several months later). If the system knew how to recover from this
> > > automatically it would howe
Re: [yocto] [PATCH] [yocto-ab-helper] Add qemux86, qemux86-64 WIC testimage buildset-config
Hi Richard Fundamentally we are performing the QA release for Yocto with this buildset config covering qemux86, qemux86-64 on core-image-lsb-sdk image. If the community feels that we have sufficient coverage which is required, we can omit this support from future QA cycles. If this is still "good to have" I will resend a "better description" in the next submission. Cheers, Aaron -Original Message- From: richard.pur...@linuxfoundation.org [mailto:richard.pur...@linuxfoundation.org] Sent: Tuesday, June 26, 2018 6:30 PM To: Chan, Aaron Chun Yew ; yocto@yoctoproject.org; Yeoh, Ee Peng Subject: Re: [PATCH] [yocto-ab-helper] Add qemux86, qemux86-64 WIC testimage buildset-config On Tue, 2018-06-26 at 16:51 +0800, Aaron Chan wrote: > Signed-off-by: Aaron Chan > --- > config.json | 32 +++- > 1 file changed, 31 insertions(+), 1 deletion(-) This patch looks like it might be correct. I say "might" as your commit message tells me what you did (kind of) but not why. I say "kind of" as you're using core-image-lsb-sdk and poky-lsb here, probably for a good reason but you don't say why. The key thing I need to know is what extra testing this provides that isn't already covered elsewhere on the autobuilder - why do we need to spend the time testing this? Can you resend with a better description of why we need to do this please? Cheers, Richard > -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH 1/2] [yocto-ab-helper] Add qemux86, qemux86-64 WIC testimage buildset-config
Hi Richard,
Kindly ignore this patch as it contains the old description. I'll send a new
patch indicating the reason behind this buildset-config is needed.
Cheers,
Aaron
From: Chan, Aaron Chun Yew
Sent: Monday, July 02, 2018 6:30 PM
To: richard.pur...@linuxfoundation.org; yocto@yoctoproject.org
Cc: Chan, Aaron Chun Yew
Subject: [PATCH 1/2] [yocto-ab-helper] Add qemux86, qemux86-64 WIC testimage
buildset-config
Signed-off-by: Aaron Chan
---
config.json | 32 +++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/config.json b/config.json
index c9dc21e..3c1f989 100644
--- a/config.json
+++ b/config.json
@@ -383,6 +383,36 @@
],
"step1" : {
"MACHINE" : "qemux86",
+"SDKMACHINE" : "x86_64",
+"DISTRO" : "poky-lsb",
+"BBTARGETS" : "wic-tools core-image-lsb-sdk",
+"EXTRACMDS" : [
+"wic create directdisk -e core-image-lsb-sdk -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-lsb-sdk/",
+"wic create directdisk-gpt -e core-image-lsb-sdk -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-lsb-sdk/",
+"wic create mkefidisk -e core-image-lsb-sdk -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-lsb-sdk/"
+],
+"extravars" : [
+"IMAGES_FSTYPES += ' wic'"
+],
+"SANITYTARGETS" : "core-image-lsb-sdk:do_testimage"
+},
+"step2" : {
+"MACHINE" : "qemux86-64",
+"SDKMACHINE" : "x86_64",
+"DISTRO" : "poky-lsb",
+"BBTARGETS" : "wic-tools core-image-lsb-sdk",
+"EXTRACMDS" : [
+"wic create directdisk -e core-image-lsb-sdk -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86-64/directdisk/core-image-lsb-sdk/",
+"wic create directdisk-gpt -e core-image-lsb-sdk -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86-64/directdisk/core-image-lsb-sdk/",
+"wic create mkefdisk -e core-image-lsb-sdk -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86-64/directdisk/core-image-lsb-sdk/"
+],
+"extravars" : [
+"IMAGES_FSTYPES += ' wic'"
+],
+"SANITYTARGETS" : "core-image-lsb-sdk:do_testimage"
+},
+"step3" : {
+"MACHINE" : "qemux86",
"BBTARGETS" : "wic-tools core-image-sato",
"EXTRACMDS" : [
"wic create directdisk -e core-image-sato -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-sato/",
@@ -390,7 +420,7 @@
"wic create mkefidisk -e core-image-sato -o
${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-sato/"
]
},
-"step2" : {
+"step4" : {
"MACHINE" : "genericx86",
"BBTARGETS" : "wic-tools core-image-sato",
"EXTRACMDS" : [
--
2.7.4
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-ab-helper] Fix syntax load config.json clobber buildStep
Hi Richard,
Added a patch fix on this on
https://lists.yoctoproject.org/pipermail/yocto/2018-July/041644.html
This resolves the handling unicode string during loadconfig() apart from
dictonary data expansions.
Let me know what are your thoughts on this. Thanks.
Cheers,
Aaron
From: richard.pur...@linuxfoundation.org [richard.pur...@linuxfoundation.org]
Sent: Tuesday, June 26, 2018 6:20 PM
To: Chan, Aaron Chun Yew; yocto@yoctoproject.org
Subject: Re: [PATCH] [yocto-ab-helper] Fix syntax load config.json clobber
buildStep
On Tue, 2018-06-26 at 15:05 +0800, Aaron Chan wrote:
> Signed-off-by: Aaron Chan
> ---
> config.json| 5 ++---
> janitor/clobberdir | 3 +--
> 2 files changed, 3 insertions(+), 5 deletions(-)
>
> diff --git a/config.json b/config.json
> index ecfca51..c9dc21e 100644
> --- a/config.json
> +++ b/config.json
> @@ -8,15 +8,14 @@
> "BUILD_HISTORY_DIRECTPUSH" : ["poky:morty", "poky:pyro",
> "poky:rocko", "poky:master"],
> "BUILD_HISTORY_FORKPUSH" : {"poky-contrib:ross/mut" :
> "poky:master", "poky:master-next" : "poky:master"},
>
> -"REPO_STASH_DIR" : "${BASE_HOMEDIR}/git/mirror",
> -"TRASH_DIR" : "${BASE_HOMEDIR}/git/trash",
> +"REPO_STASH_DIR" : "/git/mirror",
> +"TRASH_DIR" : "/git/trash",
>
> "QAMAIL_TO" : "richard.pur...@linuxfoundation.org",
> "QAMAIL_TO1" : "yocto@yoctoproject.org",
> "QAMAIL_CC1" : "pi...@toganlabs.com, ota...@ossystems.com.br, yi
> .z...@windriver.com, tracy.gray...@intel.com, joshua.g.l...@intel.com
> , apoorv.san...@intel.com, ee.peng.y...@intel.com, aaron.chun.yew.cha
> n...@intel.com, rebecca.swee.fun.ch...@intel.com, chin.huat@intel.co
> m",
> "WEBPUBLISH_DIR" : "${BASE_SHAREDDIR}/",
> "WEBPUBLISH_URL" : "https://autobuilder.yocto.io/";,
> -
> "defaults" : {
> "NEEDREPOS" : ["poky"],
> "DISTRO" : "poky",
> diff --git a/janitor/clobberdir b/janitor/clobberdir
> index 5dab5af..73ec87c 100755
> --- a/janitor/clobberdir
> +++ b/janitor/clobberdir
> @@ -19,7 +19,6 @@ import utils
>
> ourconfig = utils.loadconfig()
>
> -
> def mkdir(path):
> try:
> os.makedirs(path)
> @@ -43,7 +42,7 @@ if "TRASH_DIR" not in ourconfig:
> print("Please set TRASH_DIR in the configuration file")
> sys.exit(1)
>
> -trashdir = ourconfig["TRASH_DIR"]
> +trashdir = ourconfig["BASE_HOMEDIR"] + ourconfig["TRASH_DIR"]
You are correct there is a bug here but reverting my change to
config.json is not the way to fix this.
Have a look at this commit:
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/commit/?id=d6253df2bc21752bc0b53202e491140b0994ff63
and then see if you can send me the patch which fixes this in line with
the commit above.
Cheers,
Richard
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-ab-helper] utils.py: Resolved unicode data expansion
Hello Richard,
This morning I had the new autobuilder setup from scratch with the latest patch
you just check out, thanks for that and rollout the fix below manually.
Everything else is clean and I did what you asked me to.
However this did not resolved the data expansion when called
utils.getconfig("REPO_STASH_DIR, ourconfig), this applies when you invoke
ourconfig["REPO_STASH_DIR"]. Both yields the same errors.
We assume the JSON dumps are properly handled in ourconfig[c] when we handles
config[c] but that is not the case. I do see there is a growing issues as your
strategy to use nested JSON, however
we wont be able to handle all of these conditions needed when nested JSON
becomes complex. Anyway, i'll leave it you decide what will be best course of
action.
STDERR logs on autobuilder: (poky-tiny)
--
mv: cannot move '/home/pokybuild/yocto-worker/poky-tiny/' to a subdirectory of
itself, '${BASE_HOMEDIR}/git/mirror/1530669213-56172/poky-tiny'
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder-helper/janitor/clobberdir", line 52,
in
subprocess.check_call(['mv', x, trashdest])
File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mv',
'/home/pokybuild/yocto-worker/poky-tiny/',
u'${BASE_HOMEDIR}/git/mirror/1530669213-56172']' returned non-zero exit status 1
also, this action causes directory named ${BASE_HOMEDIR} to be created under
~/yocto-worker/poky-tiny/build
This patch
[https://lists.yoctoproject.org/pipermail/yocto/2018-July/041685.html] which
submitted today resolves the Step 1: Clobber build dir on autobuilder.
Best wishes,
Aaron
From: richard.pur...@linuxfoundation.org [richard.pur...@linuxfoundation.org]
Sent: Tuesday, July 03, 2018 9:25 PM
To: Chan, Aaron Chun Yew; yocto@yoctoproject.org
Subject: Re: [PATCH] [yocto-ab-helper] utils.py: Resolved unicode data expansion
On Tue, 2018-07-03 at 09:44 +0800, Aaron Chan wrote:
> Updated patch to trigger handlestr() when unicode string is found
> during iteration json.loads(config.json). Unicode and list with data
> expansion were not handled hence adding this patch to handle
> conversion.
> Added a debug message to dump pretty json data populated to
> ourconfig[c].
>
> e.g "REPO_STASH_DIR" read as ${BASE_HOMEDIR}/git/mirror, where it
> should be
> "REPO_STASH_DIR" as /home/pokybuild/git/mirror
>
> Signed-off-by: Aaron Chan
> ---
> scripts/utils.py | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
It took me a while to figure out why you were doing this.
We can't expand the data half way through loading the json file as
other pieces of data may later override the values. We therefore have
to defer expansion of variables until the file is completely loaded.
We therefore have to expand the variables later on, when we read them.
I pointed you at this commit:
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/commit/?id=d6253df2bc21752bc0b53202e491140b0994ff63
which changes direct accesses into ourconfig, e.g.:
ourconfig["REPO_STASH_DIR"]
into accesses using a function:
utils.getconfig("REPO_STASH_DIR", ourconfig)
and that function handles the expansion.
You should therefore be able to fix the clobberdir issue by using the
getconfig() method instead of direct access?
Cheers,
Richard
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-ab-helper] utils.py: Resolved unicode data expansion
Hi Richard,
This is what I did on which you asked me to do
~/yocto-autobuilder-helper/janitor/clobberdir
1. Change to use python3
#!/usr/bin/env/python2 to #!/usr/bin/env/python3
Retain and add print to debug message
trashdir = ourconfig["TRASH_DIR"]
print(trashdir)
Results:
mv: cannot move '/home/pokybuild/yocto-worker/nightly-x86-64-bsp/' to a
subdirectory of itself,
'${BASE_HOMEDIR}/git/trash/1530767509-20564/nightly-x86-64-bsp'
${BASE_HOMEDIR}/git/trash <--- debug message
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder-helper/janitor/clobberdir", line 54,
in
subprocess.check_call(['mv', x, trashdest])
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mv',
'/home/pokybuild/yocto-worker/nightly-x86-64-bsp/',
'${BASE_HOMEDIR}/git/trash/1530767509-20564']' returned non-zero exit status 1
program finished with exit code 1
2. Change to use python3
#!/usr/bin/env python2 to #!/usr/bin/env/python3
Change and print to debug message
trashdir = utils.getconfig("TRASH_DIR", ourconfig)
print(trashdir)
Results:
using PTY: False
/home/pokybuild/git/trash <--- debug message
program finished with exit code 0
elapsedTime=0.050781
I'll updated the patch on Item 2 in
https://lists.yoctoproject.org/pipermail/yocto/2018-July/041694.html
Thanks,
Best wishes,
Aaron
From: richard.pur...@linuxfoundation.org [richard.pur...@linuxfoundation.org]
Sent: Wednesday, July 04, 2018 3:51 PM
To: Chan, Aaron Chun Yew; yocto@yoctoproject.org
Subject: Re: [PATCH] [yocto-ab-helper] utils.py: Resolved unicode data expansion
Hi Aaron,
On Wed, 2018-07-04 at 06:42 +, Chan, Aaron Chun Yew wrote:
> This morning I had the new autobuilder setup from scratch with the
> latest patch you just check out, thanks for that and rollout the fix
> below manually. Everything else is clean and I did what you asked me
> to.
> However this did not resolved the data expansion when called
> utils.getconfig("REPO_STASH_DIR, ourconfig), this applies when you
> invoke ourconfig["REPO_STASH_DIR"]. Both yields the same errors.
> We assume the JSON dumps are properly handled in ourconfig[c] when
> we handles config[c] but that is not the case. I do see there is a
> growing issues as your strategy to use nested JSON, however
> we wont be able to handle all of these conditions needed when nested
> JSON becomes complex. Anyway, i'll leave it you decide what will be
> best course of action.
I had another look at this as we shouldn't be seeing unicode items in
this in python3. I've realised that clobberdir is using python2. If you
change clobberdir to use python3 does this mean the unicode part of the
patch is no longer needed? I think that may be the real problem!
Cheers,
Richard
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-ab-helper] clobberdir: Fix Unicode data expansion with utils API
My apologizes to that as my local copy contains the fixes from the previous
commits. Therefore this commit builts on top of it and only contains the delta
on the current changes, this is the reason why its not complete.
Thanks again for the merge.
-Original Message-
From: richard.pur...@linuxfoundation.org
[mailto:richard.pur...@linuxfoundation.org]
Sent: Thursday, July 5, 2018 5:54 PM
To: Chan, Aaron Chun Yew ; yocto@yoctoproject.org
Subject: Re: [PATCH] [yocto-ab-helper] clobberdir: Fix Unicode data expansion
with utils API
On Thu, 2018-07-05 at 13:34 +0800, Aaron Chan wrote:
> This fix is to move clobberdir from python2 to python3 to resolve
> unicode data in python2 and change the data extraction expansion from
> ourconfig["TRASH_DIR"] to utils.getconfig("TRASH_DIR", ourconfig) on
> "Clobber build dir"
> BuildStep
>
> Signed-off-by: Aaron Chan
> ---
> janitor/clobberdir | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/janitor/clobberdir b/janitor/clobberdir index
> 5e04ed7..b05a876 100755
> --- a/janitor/clobberdir
> +++ b/janitor/clobberdir
> @@ -1,4 +1,4 @@
> -#!/usr/bin/env python2
> +#!/usr/bin/env python3
> #
> # Delete a directory using the ionice backgrounded command #
At this point I think we're all getting frustrated with this. Please can you
give patches a sanity check when you're sending them. You mention in the commit
message what you need to do but the getconfig() change is missing from the
patch itself. This has happened with several previous patches too where there
were pieces missing. I deal with a lot of patches and I can't fix up each one.
The commit message mentions its fixing something but not what (a regression
introduced by a previous change).
In the interests of resolving this I've fixed up this commit and merged
it:
http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder-helper/commit/?id=54f848380fc77a9b9523bd85cd1cdce075bfb96e
Cheers,
Richard
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-ab-helper] scripts/run-jinja-parser: Add Jinja2 parser extension in autobuilder
Hi Richard,
Let me do a quick explanation on what I intended to do with this script.
1. I don't have much lava knowledge but at a quick glance this may be ok, apart
from one of the file names which I've commented on below.
[Ans]: Basically this jinja template is mapped to lava.py module where we
defines job configuration for several architectures (ARM, MIPS, PPC, x86).
lava/device/bsp-packages.jinja2 maybe we can name it to
"lava-yaml-template.jinja2" since this is YAML template for LAVA? I'm open for
all suggestions.
2. I'm not sure we want to call this "lava.py". Would something like
"lava-conf-intelqa-minnowboard.py" be a better description (as that is what it
appears to be)?
[Ans]: So I've put the LAVA job config into lava.py, the concept is very
similar to our yoctoabb (config.py)
If we change the name from lava.py to
lava-conf-intelqa-minnowboard.py, in the code we need to change to "import
lava-conf-intelqa-minnow" instead of "import lava".
I am good with the rename but it will be just long.
3. Should configuration files be places somewhere outside scripts?
[Ans]: So, basically my script is able to load the module/config outside (in
any file structure) and I have it tested like the examples below:
$ run-jinja-parser $2 $3 $4 $5
e.g. It works in this case:
~/yocto-autobuilder/lava.py
$ run-jinja-parser "~/yocto-autobuilder-helper" $2 $3 $4 $5
Or
~/yocto-autobuilder/scripts/lava.py
$ run-jinja-parser "~/yocto-autobuilder-helper/scripts" $2 $3 $4 $5
Let me know your thoughs about these. Once these changes are added, the
community will benefit from using Yocto autobuilder to trigger
their automated BSP test (using LAVA) on their platform starting with
benchmarking own manual BSP test case(s).
Cheers,
Aaron
-Original Message-
From: richard.pur...@linuxfoundation.org
[mailto:richard.pur...@linuxfoundation.org]
Sent: Friday, July 6, 2018 9:30 PM
To: Chan, Aaron Chun Yew ; yocto@yoctoproject.org
Subject: Re: [PATCH] [yocto-ab-helper] scripts/run-jinja-parser: Add Jinja2
parser extension in autobuilder
On Fri, 2018-07-06 at 17:15 +0800, Aaron Chan wrote:
> This patch is introduced as a feature in 2.6 M2 to support the
> extension of autobuilder to LAVA (Linaro Automated Validation
> Architecture).
> run-jinja2-parser loads lava config module and generates LAVA job
> config in a YAML format before its triggers LAVA server to execute a
> task.
I don't have much lava knowledge but at a quick glance this may be ok, apart
from one of the file names which I've commented on below.
> Signed-off-by: Aaron Chan
> ---
> lava/device/bsp-packages.jinja2 | 43 ++
> scripts/lava.py | 76
>
> scripts/run-jinja-parser| 97
> +
> 3 files changed, 216 insertions(+)
> create mode 100644 lava/device/bsp-packages.jinja2 create mode
> 100644 scripts/lava.py create mode 100755 scripts/run-jinja-parser
>
> diff --git a/lava/device/bsp-packages.jinja2
> b/lava/device/bsp-packages.jinja2 new file mode 100644 index
> 000..61fbcad
> --- /dev/null
> +++ b/lava/device/bsp-packages.jinja2
> @@ -0,0 +1,43 @@
> +device_type: {{ device_type }}
> +job_name: {{ job_name }}
> +timeouts:
> + job:
> +minutes: {{ timeout.job.minutes }}
> + action:
> +minutes: {{ timeout.action.minutes }}
> + connection:
> +minutes: {{ timeout.connection.minutes }}
> +priority: {{ priority }}
> +visibility: {{ visibility }}
> +actions:
> +- deploy:
> +timeout:
> + minutes: {{ deploy.timeout }}
> +to: {{ deploy.to }}
> +kernel:
> + url: {{ deploy.kernel.url }}
> + type: {{ deploy.kernel.type }}
> +modules:
> + url: {{ deploy.modules.url }}
> + compression: {{ deploy.modules.compression }}
> +nfsrootfs:
> + url: {{ deploy.nfsrootfs.url }}
> + compression: {{ deploy.nfsrootfs.compression }}
> +os: {{ deploy.os }}
> +- boot:
> +timeout:
> + minutes: {{ boot.timeout }}
> +method: {{ boot.method }}
> +commands: {{ boot.commands }}
> +auto_login: { login_prompt: {{ boot.auto_login.login_prompt }},
> username: {{ boot.auto_login.username }} }
> +prompts:
> + - {{ boot.prompts }}
> +- test:
> +timeout:
> + minutes: {{ test.timeout }}
> +name: {{ test.name }}
> +definitions:
> +- repository: {{ test.definitions.repository }}
> + from: {{ test.definitions.from }}
> + path: {{ test.definitions.path }}
> + name: {{ test.definitions.name }}
> diff --git a/scripts/lava.py b/scripts/lava.py new file mode 100644
Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN
My name is Aaron and not Aron for start
Martin,
Please try this
#!/usr/bin/env python2
import os
a = os.path.join('http://', 'alibaba.com')
b = '/'.join(['http://', 'alibaba.com'])
c = '/'.join(['http:/', 'alibaba.com', ''])
print(a)
print(b)
print(c)
and repeat the same for python3, I got the following results:
http://alibaba.com
http:///alibaba.com
http://alibaba.com/
Cheers,
Aaron
____
From: Martin Hundebøll [mar...@geanix.com]
Sent: Tuesday, July 10, 2018 2:10 PM
To: Chan, Aaron Chun Yew; richard.pur...@linuxfoundation.org;
yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults
autobuilder URL based on FQDN
Hi Aron,
On 2018-07-10 05:18, Aaron Chan wrote:
> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
> The socket module allows the retrieval on FQDN and constructs the entire
> URL by default, this default settings can be overwritten in c['buildbotURL']
> based on local administrator preferences.
>
> Signed-off-by: Aaron Chan
> ---
> master.cfg | 7 +--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/master.cfg b/master.cfg
> index fca80d2..49ddeb4 100644
> --- a/master.cfg
> +++ b/master.cfg
> @@ -4,6 +4,7 @@
> import os
> import imp
> import pkg_resources
> +import socket
>
> from buildbot.plugins import *
> from buildbot.plugins import db
> @@ -55,6 +56,7 @@ imp.reload(services)
> imp.reload(www)
>
> c = BuildmasterConfig = {}
> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port)
> + '/')
Why use `os.path.join()` here? It isn't supposed to be used to construct
url's, and is overkill for this case, and you'd end up with "http:///...";.
// Martin
>
> # Disable usage reporting
> c['buildbotNetUsageData'] = None
> @@ -76,6 +78,7 @@ c['www'] = www.www
> c['workers'] = workers.workers
>
> c['title'] = "Yocto Autobuilder"
> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/";
> +c['titleURL'] = url
> # visible location for internal web server
> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
> +c['buildbotURL'] = url
>
--
Kind regards,
Martin Hundebøll
Embedded Linux Consultant
+45 61 65 54 61
mar...@geanix.com
Geanix IVS
DK39600706
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN
Hi Martin,
My initial concern was that `os.path.join()` was meant for OS
independent path concatenation, and not constructing URLs.
My two first points still stand, though... Why not just
string-concatenate the separate parts; i.e.
url = 'http://' + socket.getfqdn() + ':' + str(config.web_port) + '/'
[Reply] You have a valid point, however in this case seems like os.path.join()
doesn't seem to construct a path. It's a bug.
Will send a new patch on this one. Thanks for reviewing it.
Cheers,
Aaron
From: Martin Hundebøll [mar...@geanix.com]
Sent: Tuesday, July 10, 2018 3:14 PM
To: Chan, Aaron Chun Yew; richard.pur...@linuxfoundation.org;
yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults
autobuilder URL based on FQDN
Hi Aaron,
On 2018-07-10 08:59, Chan, Aaron Chun Yew wrote:
> My name is Aaron and not Aron for start
Sorry about that.
> Martin,
>
> Please try this
>
> #!/usr/bin/env python2
>
> import os
>
> a = os.path.join('http://', 'alibaba.com')
> b = '/'.join(['http://', 'alibaba.com'])
> c = '/'.join(['http:/', 'alibaba.com', ''])
>
> print(a)
> print(b)
> print(c)
>
> and repeat the same for python3, I got the following results:
>
> http://alibaba.com
> http:///alibaba.com
> http://alibaba.com/
I see, thanks for clarifying.
My initial concern was that `os.path.join()` was meant for OS
independent path concatenation, and not constructing URLs.
My two first points still stand, though... Why not just
string-concatenate the separate parts; i.e.
url = 'http://' + socket.getfqdn() + ':' + str(config.web_port) + '/'
But I won't stand in the way of the patch for such a stylish point, so
feel free to ignore it :)
// Martin
> Cheers,
> Aaron
>
> From: Martin Hundebøll [mar...@geanix.com]
> Sent: Tuesday, July 10, 2018 2:10 PM
> To: Chan, Aaron Chun Yew; richard.pur...@linuxfoundation.org;
> yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
> Subject: Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults
> autobuilder URL based on FQDN
>
> Hi Aron,
>
> On 2018-07-10 05:18, Aaron Chan wrote:
>> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
>> The socket module allows the retrieval on FQDN and constructs the entire
>> URL by default, this default settings can be overwritten in c['buildbotURL']
>> based on local administrator preferences.
>>
>> Signed-off-by: Aaron Chan
>> ---
>>master.cfg | 7 +--
>>1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/master.cfg b/master.cfg
>> index fca80d2..49ddeb4 100644
>> --- a/master.cfg
>> +++ b/master.cfg
>> @@ -4,6 +4,7 @@
>>import os
>>import imp
>>import pkg_resources
>> +import socket
>>
>>from buildbot.plugins import *
>>from buildbot.plugins import db
>> @@ -55,6 +56,7 @@ imp.reload(services)
>>imp.reload(www)
>>
>>c = BuildmasterConfig = {}
>> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port)
>> + '/')
>
> Why use `os.path.join()` here? It isn't supposed to be used to construct
> url's, and is overkill for this case, and you'd end up with "http:///...";.
>
> // Martin
>
>>
>># Disable usage reporting
>>c['buildbotNetUsageData'] = None
>> @@ -76,6 +78,7 @@ c['www'] = www.www
>>c['workers'] = workers.workers
>>
>>c['title'] = "Yocto Autobuilder"
>> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/";
>> +c['titleURL'] = url
>># visible location for internal web server
>> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
>> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
>> +c['buildbotURL'] = url
>>
>
> --
> Kind regards,
> Martin Hundebøll
> Embedded Linux Consultant
>
> +45 61 65 54 61
> mar...@geanix.com
>
> Geanix IVS
> DK39600706
>
--
Kind regards,
Martin Hundebøll
Embedded Linux Consultant
+45 61 65 54 61
mar...@geanix.com
Geanix IVS
DK39600706
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view
Hi Richard, [Richard] I think this means you're using python2 and we really should be using python3 as I don't want to support both... [Reply] This error/bug was found during buildbot startup meaning this is out of my control. Maybe you have a fix for this, otherwise I do suggest to add in __init__.py in places where we need to source custom modules. $ buildbot start ~/yocto-controller $ cat -n 100 ~/yocto-controller/twistd.log 58002018-07-10 18:42:13+0800 [-] Main loop terminated. 5801 2018-07-10 18:42:13+0800 [-] Server Shut Down. 5802 2018-07-10 18:42:27+0800 [-] Loading buildbot.tac... 5803 2018-07-10 18:42:27+0800 [-] Loaded. 5804 2018-07-10 18:42:27+0800 [-] twistd 18.4.0 (/usr/bin/python 2.7.13) starting up. 5805 2018-07-10 18:42:27+0800 [-] reactor class: twisted.internet.epollreactor.EPollReactor. 5806 2018-07-10 18:42:27+0800 [-] Starting BuildMaster -- buildbot.version: 1.2.0 5807 2018-07-10 18:42:27+0800 [-] Loading configuration from '/home/ab/yocto-controller/master.cfg' 5808 2018-07-10 18:42:27+0800 [-] error while parsing config file: 5809 Traceback (most recent call last): 5810File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 266, in 5811 inContext.theWork = lambda: context.call(ctx, func, *args, **kw) 5812File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext 5813 return self.currentContext().callWithContext(ctx, func, *args, **kw) 5814File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext 5815 return func(*args,**kw) 5816File "/home/ab/.local/lib/python2.7/site-packages/buildbot/config.py", line 182, in loadConfig 5817 self.basedir, self.configFileName) 5818 --- --- 5819File "/home/ab/.local/lib/python2.7/site-packages/buildbot/config.py", line 140, in loadConfigDict 5820 execfile(filename, localDict) 5821File "/home/ab/.local/lib/python2.7/site-packages/twisted/python/compat.py", line 246, in execfile 5822 exec(code, globals, locals) 5823File "/home/ab/yocto-controller/master.cfg", line 11, in 5824 from yoctoabb import builders, config, schedulers, workers, services, www 5825 exceptions.ImportError: No module named yoctoabb 5826 5827 2018-07-10 18:42:27+0800 [-] Configuration Errors: 5828 2018-07-10 18:42:27+0800 [-] error while parsing config file: No module named yoctoabb (traceback in logfile) 5829 2018-07-10 18:42:27+0800 [-] Halting master. 5830 2018-07-10 18:42:27+0800 [-] BuildMaster startup failed 5831 2018-07-10 18:42:27+0800 [-] BuildMaster is stopped 5832 2018-07-10 18:42:27+0800 [-] Main loop terminated. 5833 2018-07-10 18:42:27+0800 [-] Server Shut Down. Cheers, Aaron From: richard.pur...@linuxfoundation.org [richard.pur...@linuxfoundation.org] Sent: Tuesday, July 10, 2018 4:40 PM To: Chan, Aaron Chun Yew; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul Subject: Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view On Tue, 2018-07-10 at 11:24 +0800, Aaron Chan wrote: > This patch is to fix the inconsistency in loading custom module > yoctoabb & yocto_console_view during Buildbot Master startup. > > Signed-off-by: Aaron Chan > --- > __init__.py| 0 > yocto_console_view/__init__.py | 0 > 2 files changed, 0 insertions(+), 0 deletions(-) > create mode 100644 __init__.py > create mode 100644 yocto_console_view/__init__.py I think this means you're using python2 and we really should be using python3 as I don't want to support both... Cheers, Richard -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL based on FQDN
Hi Richard,
[Richard] I appreciate what you're trying to do here and make this
autoconfigure.
Unfortunately the urls can't always be figured out this way, the above
for example drops the "/main/" part, without which the autobuilder
won't work. The server can be behind forwarding or proxies/firewalls
which also make this problematic.
I also agree with Martin that using os.path.join() in a url is
incorrect, its not meant for urls.
Most people setting up an autobuilder will need to have some
customisations on top of yocto-autobuilder2, I don't think its possible
to avoid that. I'd therefore perhaps try and concentrate on having key
modules like the lava integration available and accept there will
always be some local configuration the end user needs to make to things
like the URL details? Even the main autobuilder adds in passwords and
some other local config on top of the stardard repo...
[Reply] I do agree with you that some customization are required to be built on
top of autobuilder.
However this patch is submitted because the URL link below is not
working on my end and
only http://localhost:8010/ is working without the need to change
the master.cfg
So therefore, due to several rounds of commission i decided to send
a patch which defaults
URL following the FQDN sets on the host machines.
c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
c['buildbotURL'] = "http://localhost:8010/'
The intention of this patch is to reduce the need for local configurations, yes
I do agree that there will
be some level of customization needed on local setup. I'll leave it to you then
to determine whats best.
Cheers,
Aaron
From: richard.pur...@linuxfoundation.org [richard.pur...@linuxfoundation.org]
Sent: Tuesday, July 10, 2018 4:55 PM
To: Chan, Aaron Chun Yew; yocto@yoctoproject.org; Burton, Ross; Eggleton, Paul
Subject: Re: [PATCH] [yocto-autobuilder] master.cfg: Defaults autobuilder URL
based on FQDN
On Tue, 2018-07-10 at 11:18 +0800, Aaron Chan wrote:
> This patch is to enable auto-assignments buildbot URL based on Hosts FQDN.
> The socket module allows the retrieval on FQDN and constructs the entire
> URL by default, this default settings can be overwritten in c['buildbotURL']
> based on local administrator preferences.
>
> Signed-off-by: Aaron Chan
> ---
> master.cfg | 7 +--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/master.cfg b/master.cfg
> index fca80d2..49ddeb4 100644
> --- a/master.cfg
> +++ b/master.cfg
> @@ -4,6 +4,7 @@
> import os
> import imp
> import pkg_resources
> +import socket
>
> from buildbot.plugins import *
> from buildbot.plugins import db
> @@ -55,6 +56,7 @@ imp.reload(services)
> imp.reload(www)
>
> c = BuildmasterConfig = {}
> +url = os.path.join('http://', socket.getfqdn() + ':' + str(config.web_port)
> + '/')
>
> # Disable usage reporting
> c['buildbotNetUsageData'] = None
> @@ -76,6 +78,7 @@ c['www'] = www.www
> c['workers'] = workers.workers
>
> c['title'] = "Yocto Autobuilder"
> -c['titleURL'] = "https://autobuilder.yoctoproject.org/main/";
> +c['titleURL'] = url
> # visible location for internal web server
> -c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
> +# - Default c['buildbotURL'] = "https://autobuilder.yoctoproject.org/main/";
> +c['buildbotURL'] = url
I appreciate what you're trying to do here and make this autoconfigure.
Unfortunately the urls can't always be figured out this way, the above
for example drops the "/main/" part, without which the autobuilder
won't work. The server can be behind forwarding or proxies/firewalls
which also make this problematic.
I also agree with Martin that using os.path.join() in a url is
incorrect, its not meant for urls.
Most people setting up an autobuilder will need to have some
customisations on top of yocto-autobuilder2, I don't think its possible
to avoid that. I'd therefore perhaps try and concentrate on having key
modules like the lava integration available and accept there will
always be some local configuration the end user needs to make to things
like the URL details? Even the main autobuilder adds in passwords and
some other local config on top of the stardard repo...
Cheers,
Richard
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view
Hi Richard, It appears that your using virtualenv (switch to python3) to startup your buildbot processes and not using the default python from host distribution. I am aware that you can switch interpreter to python2/3 in virtualenv. Shouldn’t we handle these for both in events users uses the default host distribution as default python2 and still make it worked? What is the best practices here and should all the buildbot-workers start in virtualenv defaults to python3 as well ? I am little lost right here, please advise. Cheers, Aaron Try using virtualenv: $ python --version Python 2.7.15rc1 $ virtualenv -p python3 test Already using interpreter /usr/bin/python3 Using base prefix '/usr' New python executable in /media/build1/test/bin/python3 Also creating executable in /media/build1/test/bin/python Installing setuptools, pkg_resources, pip, wheel...done. $ . ./test/bin/activate (test) richard@dax:/media/build1/$ python --version Python 3.6.5 If you start buildbot within the virtualenv, you should see it using python3. Cheers, Richard -Original Message- From: richard.pur...@linuxfoundation.org [mailto:richard.pur...@linuxfoundation.org] Sent: Wednesday, July 11, 2018 6:17 AM To: Chan, Aaron Chun Yew ; yocto@yoctoproject.org; Burton, Ross ; Eggleton, Paul Subject: Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view On Tue, 2018-07-10 at 10:47 +0000, Chan, Aaron Chun Yew wrote: > [Richard] I think this means you're using python2 and we really should > be using > python3 as I don't want to support both... > > [Reply] This error/bug was found during buildbot startup meaning this > is out of my control. No, it is not out of your control. > Maybe you have a fix for this, otherwise I do suggest to > add in __init__.py in places where we need to source custom modules. > > $ buildbot start ~/yocto-controller > $ cat -n 100 ~/yocto-controller/twistd.log Try using virtualenv: $ python --version Python 2.7.15rc1 $ virtualenv -p python3 test Already using interpreter /usr/bin/python3 Using base prefix '/usr' New python executable in /media/build1/test/bin/python3 Also creating executable in /media/build1/test/bin/python Installing setuptools, pkg_resources, pip, wheel...done. $ . ./test/bin/activate (test) richard@dax:/media/build1/$ python --version Python 3.6.5 If you start buildbot within the virtualenv, you should see it using python3. Cheers, Richard -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view
Hi Richard, Thanks for the well crafted explanation, and after talking to Paul I got a clearer picture on which direction we should be heading and we should align and work towards python3. Maybe I was not aware, my bad. Cheers, Aaron -Original Message- From: richard.pur...@linuxfoundation.org [mailto:richard.pur...@linuxfoundation.org] Sent: Wednesday, July 11, 2018 3:12 PM To: Chan, Aaron Chun Yew ; yocto@yoctoproject.org; Burton, Ross ; Eggleton, Paul Subject: Re: [PATCH] [yocto-autobuilder] init: Fix the import module yoctoabb & yocto_console_view On Wed, 2018-07-11 at 02:23 +0000, Chan, Aaron Chun Yew wrote: > It appears that your using virtualenv (switch to python3) to startup > your buildbot processes and not using the default python from host > distribution. > I am aware that you can switch interpreter to python2/3 in virtualenv. > Shouldn’t we handle these for both in events users uses the default > host distribution as default python2 and still make it worked? What is > the best practices here and should all the buildbot- workers start in > virtualenv defaults to python3 as well ? I am little lost right here, > please advise. There are multiple ways to achieve this, you can make /usr/bin/python point at python3 too, or simply not install python2 on the controller. Using virtualenv was just a simple way to illustrate how you can use it. My worry about trying to support both python versions is that as you've found out, there are problems that can occur in one that don't occur in the other, which then means twice as much testing. Since python2 is an official "dead end" and as a project we're trying hard to use python3 everywhere, it doesn't seem to make sense to try and support python2 on new code such as the autobuilder? Cheers, Richard -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [RFC] Yocto Autobuilder and LAVA Integration
Hi Anibal/RP, > In order to do a distributed Boards testing the Yocto Autobuilder > needs to publish in some place accessible the artifacts (image, > kernel, etc) to flash/boot the board and the test suite expected to > execute. [Reply] That is correct, since Linaro have this in place to use https://archive.validation.linaro.org/directories/ and I have look into this as well, we can leverage on this but I am up for any suggestion you might have. So the idea here is that we have a placeholder to store the publish artifacts remotely and deploy using native curl command with token access. Then based on your LAVA job definitions we can instruct LAVA to source the images via https. Having said that, the deploy stage in LAVA must have some capabilities to read a token file in the LAVA job defintion and pick up the binaries from public repo (git LFS). In order for Board Distributed Tests to happen, there are 2 items in my wish lists 1. Public hosting of binary repository - with access control 2. Ease Handshaking between two(2) different systems CI (e.g. Jenkins/Autobuilder) with LAVA a. Exchange build property (metadata) - includes hardware info, system info b. Test reporting results > I created a simple LAVA test definition that allows run testimage > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to > get target and server IP addresses. [Reply] Although the lava test shell have these capabilities to use lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way we retrieve the Ipv4 address is reading the logs from LAVA thru XML-RPC and grep the pattern matching string which contains IP even before the HW get initialize entirely then parse IP back to the Yocto Autobuilder. http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs > Some of the tasks, I identified, (if is accepted) > > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , > move the EXTRA_PLAIN_CMDS to a class. > - Poky/OE: Review/fix test-export or provide other mechanism to export > the test suite. > - Poky/OE: Review/fix test-export or provide other > mechanism to export > the test suite. [Reply] I would like to understand further what is the implementation here and how it addresses the problems that we have today. I believe in the past, Tim has tried to enable testexport and transfer the testexport into the DUT but it was not very successful and we found breakage. > - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job > templates across boards. [Reply] I couldn’t be more supportive on this having a common LAVA job template across boards but I would like to stress this, we don’t exactly know how community will define their own LAVA job definition, therefore what I had in mind as per today is to create a placeholde where LAVA job templates can be define and other boards/community can reuse the same template if it fits their use cases. In general the templates we have today are created to fit into Yocto Project use cases. Lastly there are some works I've done on provisiong QEMU on LAVA sourceing from Yocto Project public releases, I am looking at where we can upstream this https://github.com/lab-github/yoctoproject-lava-test-shell Thanks! Cheers, Aaron Chan Open Source Technology Center Intel -Original Message- From: richard.pur...@linuxfoundation.org [mailto:richard.pur...@linuxfoundation.org] Sent: Thursday, November 8, 2018 6:45 AM To: Anibal Limon ; yocto@yoctoproject.org Cc: Nicolas Dechesne ; Chan, Aaron Chun Yew Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration Hi Anibal, On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote: > We know the need to execute OE testimage over real HW not only QEMU, > > I'm aware that currently there is an implementation on the Yocto > Autobuilder Helper , this initial implementation looks pretty well > separating parts for template generation [1] and the script to send > jobs to LAVA [2]. > > There are some limitations. > > - Requires that the boards are accessible trough SSH (same network?) > by the Autobuilder, so no distributed LAB testing. > - LAVA doesn't know about test results because the execution is > injected via SSH. > > In order to do a distributed Boards testing the Yocto Autobuilder > needs to publish in some place accessible the artifacts (image, > kernel, etc) to flash/boot the board and the test suite expected to > execute. > > Currently there is a func
