On Fri, 2024-07-05 at 16:46 +0200, Alexis Lothoré via lists.openembedded.org
wrote:
> From: Alexis Lothoré <alexis.loth...@bootlin.com>
>
> Fixes [YOCTO 15536]
>
> The postactions retrieval actions currently rely on scp executed
> individually on any file or directory expanded from
> TESTIMAGE_FAILED_QA_ARTIFACTS. Unfortunately, symlinks are not preserved
> with this mechanism, which lead to big storage space consumption. Things
> may go even worse if those symlinks create some circular chains. This
> mechanism then needs to be updated to preserve symlinks instead of
> following them during copy. There are multiple ways to do it:
> - create a local archive on the target and execute scp on this file
> - use rsync instead of scp for all files
> - create an archive and pipe it to ssh instead of storing it onto the
> target
>
> The first solution may create pressure on targets storage space, while the
> second assumes that rsync is installed on the target, which may not be
> true. So the third one is a compromise: tar is very likely present, at
> least through busybox, and no disk space is used on the target.
>
> Replace the current per-file scp call by a single call to tar run on the
> target. Retrieve the generated compressed archive directly from SSH output,
> and feed it to another tar process but on host, to uncompress and extract
> it at the same place as before.
>
> Signed-off-by: Alexis Lothoré <alexis.loth...@bootlin.com>
> ---
> meta/lib/oeqa/utils/postactions.py | 19 +++++++++----------
> 1 file changed, 9 insertions(+), 10 deletions(-)
>
> diff --git a/meta/lib/oeqa/utils/postactions.py
> b/meta/lib/oeqa/utils/postactions.py
> index ecdddd2d40e3..f7ce25f2af8e 100644
> --- a/meta/lib/oeqa/utils/postactions.py
> +++ b/meta/lib/oeqa/utils/postactions.py
> @@ -62,17 +62,16 @@ def get_artifacts_list(target, raw_list):
> return result
>
> def retrieve_test_artifacts(target, artifacts_list, target_dir):
> + import io, subprocess
> local_artifacts_dir = os.path.join(target_dir, "artifacts")
> - for artifact_path in artifacts_list:
> - if not os.path.isabs(artifact_path):
> - bb.warn(f"{artifact_path} is not an absolute path")
> - continue
> - try:
> - dest_dir = os.path.join(local_artifacts_dir,
> os.path.dirname(artifact_path[1:]))
> - os.makedirs(dest_dir, exist_ok=True)
> - target.copyFrom(artifact_path, dest_dir)
> - except Exception as e:
> - bb.warn(f"Can not retrieve {artifact_path} from test target:
> {e}")
> + try:
> + cmd = f"tar zcf - {" ".join(artifacts_list)}"
Thanks for the patch. The syntax above causes failures:
https://valkyrie.yoctoproject.org/#/builders/95/builds/59/steps/14/logs/stdio
I've put a fix on master-next which I can squash in but it does make me
wonder how it was tested.
Cheers,
Richard
> + (status, output) = target.run(cmd, raw = True)
> + if status != 0 or not output:
> + raise Exception("Error while fetching compressed artifacts")
> + p = subprocess.run(["tar", "zxf", "-", "-C", local_artifacts_dir],
> input=output)
> + except Exception as e:
> + bb.warn(f"Can not retrieve {artifact_path} from test target: {e}")
>
> def list_and_fetch_failed_tests_artifacts(d, tc):
> artifacts_list = get_artifacts_list(tc.target,
> d.getVar("TESTIMAGE_FAILED_QA_ARTIFACTS"))
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201618):
https://lists.openembedded.org/g/openembedded-core/message/201618
Mute This Topic: https://lists.openembedded.org/mt/107054834/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
