On 13/12/2023 20:55, Richard Purdie wrote:
On Wed, 2023-12-13 at 20:30 +0000, Eilís 'pidge' Ní Fhlannagáin wrote:
This takes the work rburton did on image screenshot testing and
expands it.
Right now this only works for qemux86-64. Some standardization
of screensize/resolution needs to happen with runqemu params for other
machines. There is an issue in qemux86-64 (and possibly others) where
the screenswitch icon is only half present. This causes the test to
fail.
This test takes a screendump of a qemu image (for now, just
core-image-sato for qemux86-64), and compares it to an image we
have on record. Some normalisation of the different qemu configs
need to happen to be able to support all machines. Example, the
qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do
the same blanking out process for the screenshot, so the images should
have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in
your bblayers.conf and the following in local.conf:
IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp "
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pi...@baylibre.com>
Co-authored-by: Ross Burton <ross.bur...@arm.com>
Co-authored-by: Eilís 'pidge' Ní Fhlannagáin <pi...@baylibre.com>
---
.../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes
meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++
2 files changed, 34 insertions(+)
create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
create mode 100644 meta/lib/oeqa/runtime/cases/login.py
diff --git a/meta/lib/oeqa/runtime/cases/login.py
b/meta/lib/oeqa/runtime/cases/login.py
new file mode 100644
index 00000000000..9e351fdeb29
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/login.py
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+ @OEHasPackage(['python3-qemu-qmp'])
This only works if we have python3-qemu-qmp in the target? I thought
the monitor was external?
+ def test_screenshot(self):
+ if self.td.get('MACHINE') != "qemux86-64":
+ self.fail
So we're going to throw failures on all other machines? Wouldn't a skip
be more appropriate? Can we provide images for other machines?
I can get images for other machines, but it's going to take a bit to run
everything. I've debug code to grab those images. The other option is to
ensure the other qemu machines have something like QB_GRAPHICS =
"-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions
Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")
+
+ if bb.utils.which(os.getenv('PATH'), "convert") is not None and
bb.utils.which(os.getenv('PATH'), "compare") is not None:
This works for nativesdk-imagemagick but it would need convert+compare
in HOSTTOOLS to work for a host binary. I'm torn on the best way to
find that otherwise.
Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.
+ with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login",
suffix=".png") as t:
+ ret = self.target.runner.run_monitor("screendump", args={"filename": t.name,
"format":"png"})
+ # Use the meta-oe version of convert, along with it's suffix
+ cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22'
{1}".format(t.name, t.name)
+ proc = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ output, error = proc.communicate()
You probably want subprocess.check_result()
check_output, but yes, that works here.... but...
+
+ # Use the meta-oe version of compare, along with it's suffix
+ cmd = "compare.im7 -metric MSE {0}
{1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name,
self.td.get('COREBASE'), self.td.get('MACHINE'))
+ proc = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ output, error = proc.communicate()
and check_output()
...doesn't here. In this case, we should probably either use run() or
Popen() because imagemagick returns non-failing output on stderr (which
is what we need) as well stdout (which we don't really want). Using
Popen allows us to do this a bit cleaner as opposed to check_output.
+ diff=float(error.decode('utf-8').replace("(",
"").replace(")","").split()[1])
+ self.assertEqual(0, diff, "Screenshot diff is %s." %
(str(diff)))
+ else:
+ self.fail
Wouldn't:
self.skipTest("could not find convert tool from imagemagick")
be better?
Right now I can't merge this. With skip()s I can at least merge whilst
we sort the other pieces.
> Cheers,
Richard
--
Eilís 'pidge' Ní Fhlannagáin
BayLibre - At the Heart of Embedded Linux
www.baylibre.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192391):
https://lists.openembedded.org/g/openembedded-core/message/192391
Mute This Topic: https://lists.openembedded.org/mt/103157570/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
