Hi RP,
Thank you very much again for continuously providing your precious feedbacks to
me.
Also thank you very much for spending great amount of time to improve this
patchset siginificantly.
I did some testing with the latest from resulttool: Update to use gitarchive
library function.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4
I had 2 questions.
1. For "resulttool regression", currently it was comparing result id set
without comprehending the difference in the host distro used to executed the
oeselftest. Example: it was matching oeselftest run with fedora28 host distro
with oeselftest run with ubuntu18 host distro, is this the expected behavior?
Match: oeselftest_fedora-28_qemux86-64_20190201181656
oeselftest_ubuntu-18.04_qemux86-64_20190201175023
Match: oeselftest_fedora-26_qemux86-64_20190131144317
oeselftest_fedora-26_qemux86-64_20190131144317
Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
oeselftest_fedora-28_qemux86-64_20190201181656
Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
oeselftest_opensuse-42.3_qemux86-64_20190126152612
I believe that we shall comprehend the 'HOST_DISTRO' configuration inside the
regression_map.
regression_map = {
- "oeselftest": ['TEST_TYPE', 'MACHINE'],
+ "oeselftest": ['TEST_TYPE', 'HOST_DISTRO', 'MACHINE'],
"runtime": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
'IMAGE_PKGTYPE', 'DISTRO'],
"sdk": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
'SDKMACHINE'],
"sdkext": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
'SDKMACHINE']
}
After comprehending this 'HOST_DISTRO', it was able to perform regression for
oeselftest with the matching host distro.
Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
oeselftest_ubuntu-18.04_qemux86-64_20190201175023
Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
oeselftest_opensuse-42.3_qemux86-64_20190126152612
Match: oeselftest_fedora-26_qemux86-64_20190131144317
oeselftest_fedora-26_qemux86-64_20190131144317
Match: oeselftest_fedora-28_qemux86-64_20190201181656
oeselftest_fedora-28_qemux86-64_20190201181656
2. For "resulttool store", I had noticed that it will now generally stored
testresults.json in a meaningful file directory structure based on the
store_map except oeselftest. oeselftest currently store multiple result id set
inside oselftest file directory without comprehend the host distro.
For example runtime, store testresult.json with the configured store_map.
├── oeselftest
│ └── testresults.json
├── runtime
│ ├── poky
│ │ ├── qemuarm
│ │ │ ├── core-image-minimal
│ │ │ │ └── testresults.json
│ │ │ ├── core-image-sato
│ │ │ │ └── testresults.json
│ │ │ └── core-image-sato-sdk
│ │ │ └── testresults.json
│ │ ├── qemuarm64
│ │ │ ├── core-image-minimal
│ │ │ │ └── testresults.json
│ │ │ ├── core-image-sato
│ │ │ │ └── testresults.json
│ │ │ └── core-image-sato-sdk
│ │ │ └── testresults.json
I believe that we shall again comprehend the 'HOST_DISTRO' configuration inside
the store_map.
store_map = {
- "oeselftest": ['TEST_TYPE'],
+ "oeselftest": ['TEST_TYPE','HOST_DISTRO'],
"runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
"sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
"sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME']
Doing so, it will store oeselftest in a more useful file directory structure
with host distro comprehended.
└── oeselftest
├── fedora-26
│ └── testresults.json
├── fedora-28
│ └── testresults.json
├── opensuse-42.3
│ └── testresults.json
└── ubuntu-18.04
└── testresults.json
Please let me know if you have any question related to above.
Best regards,
Yeoh Ee Peng
-----Original Message-----
From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org]
Sent: Monday, February 18, 2019 6:46 AM
To: Yeoh, Ee Peng <ee.peng.y...@intel.com>;
openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Sun, 2019-02-17 at 17:54 +0000, Richard Purdie wrote:
> > Despite my changes there are things that still need to be done.
> > Essential things which need to happen before this code merges:
> >
> > * oe-git-archive is importing using the commit/branch of the current
> > repo, not the data in the results file.
Also now fixed. I put my patches into master-next too.
With this working, I was able to run something along the lines of:
for D in $1/*; do
resulttool store $D $2 --allow-empty done
on the autobuilder's recent results which lead to the creation of this
repository:
http://git.yoctoproject.org/cgit.cgi/yocto-testresults/
> > * Revisit and redo the way the git branch handling is happening.
> > We
> > really want to model how oe-build-perf-report handles git repos
> > for
> > comparisons:
> > - Its able to query data from git repos without changing the
> > current
> > working branch,
> > - it can search on tag formats to find comparison data
Which means we now need to make the git branch functionality of the report and
regression commands compare with the above repo, so we're a step closer to
getting thie merged.
Ultimately we'll auto-populate the above repo by having the autobuilder run a
"store" command at the end of its runs.
I have a feeling I may have broken the resulttool selftests so that is
something else which will need to be fixed before anything merges. Time for me
to step away from the keyboard for a bit too.
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
