On Fri, Jul 02, 2021 at 11:03:52PM +0300, Alper Nebi Yasak wrote: > On 02/07/2021 22:01, Tom Rini wrote: > > Hey all, > > > > I started taking a look at moving to guestfish to see if this resolves > > the latest problem I've run in to: > > https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 > > which I think is due to guestmount not being done in time for the test. > > That failing test's setup uses virt-make-fs, different from what you're > changing below. I locally only see that failure for the clang build, and > it still fails after adding time.sleep(300) after its virt-make-fs > calls. I don't think it's an issue in the test setup.
Ah good, I need to go catch up on that thread again, thanks for looking.
> > So I started converting things to use guestfish directly:
> > diff --git a/test/py/tests/test_fs/conftest.py
> > b/test/py/tests/test_fs/conftest.py
> > index 7325486cdb1a..e8899cfdd118 100644
> > --- a/test/py/tests/test_fs/conftest.py
> > +++ b/test/py/tests/test_fs/conftest.py
> > @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config):
> > fs_ubtype = fstype_to_ubname(fs_type)
> > check_ubconfig(u_boot_config, fs_ubtype)
> >
> > - mount_dir = u_boot_config.persistent_data_dir + '/mnt'
> > + data_dir = u_boot_config.persistent_data_dir + '/data'
> >
> > - small_file = mount_dir + '/' + SMALL_FILE
> > - big_file = mount_dir + '/' + BIG_FILE
> > + small_file = data_dir + '/' + SMALL_FILE
> > + big_file = data_dir + '/' + BIG_FILE
> >
> > try:
> >
> > @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config):
> > return
> >
> > try:
> > - check_call('mkdir -p %s' % mount_dir, shell=True)
> > + check_call('mkdir -p %s' % data_dir, shell=True)
> > except CalledProcessError as err:
> > pytest.skip('Preparing mount folder failed for filesystem: ' +
> > fs_type + '. {}'.format(err))
> > call('rm -f %s' % fs_img, shell=True)
> > return
> >
> > try:
> > - # Mount the image so we can populate it.
> > - mount_fs(fs_type, fs_img, mount_dir)
> > - except CalledProcessError as err:
> > - pytest.skip('Mounting to folder failed for filesystem: ' + fs_type
> > + '. {}'.format(err))
> > - call('rmdir %s' % mount_dir, shell=True)
> > - call('rm -f %s' % fs_img, shell=True)
> > - return
> > -
> > - try:
> > - # Create a subdirectory.
> > - check_call('mkdir %s/SUBDIR' % mount_dir, shell=True)
> > -
> > - # Create big file in this image.
> > + # Create big file to copy in to the image.
> > # Note that we work only on the start 1MB, couple MBs in the 2GB
> > range
> > # and the last 1 MB of the huge 2.5GB file.
> > # So, just put random values only in those areas.
> > @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config):
> > check_call('dd if=/dev/urandom of=%s bs=1M count=1 seek=2499'
> > % big_file, shell=True)
> >
> > - # Create a small file in this image.
> > + # Create a small file to copy in to the image.
> > check_call('dd if=/dev/urandom of=%s bs=1M count=1'
> > % small_file, shell=True)
> >
> > + # Copy the files in to the image and add a subdirectory.
> > + # Create a subdirectory.
> > + check_call('guestfish add %s : run : mount /dev/sda / : mkdir
> > /SUBDIR : copy-in %s %s /'
> > + % (fs_img, big_file, small_file), shell=True)
>
> It could be faster to do things within guestfish as much as possible,
> instead of preparing the files outside and copying them in.
This is, I believe, as much as possible. You can't run arbitrary
commands via guestfish, or at least I didn't see how. "dd" isn't really
dd, for example.
> Also it looks like python bindings are available as python3-guestfs on
> Debian and Ubuntu, just not on pypi.org.
I saw there's some reason why you can't pip install them, and wasn't
sure it would be any faster, based on doing these commands outside of
pytest first.
> > # Delete the small file copies which possibly are written as part
> > of a
> > # previous test.
> > # check_call('rm -f "%s.w"' % MB1, shell=True)
> > @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config):
> >
> > except CalledProcessError as err:
> > pytest.skip('Setup failed for filesystem: ' + fs_type + '.
> > {}'.format(err))
> > - umount_fs(mount_dir)
> > return
> > else:
> > - umount_fs(mount_dir)
> > yield [fs_ubtype, fs_img, md5val]
> > finally:
> > - call('rmdir %s' % mount_dir, shell=True)
> > + call('rmdir %s' % data_dir, shell=True)
> > call('rm -f %s' % fs_img, shell=True)
> >
> > #
> >
> > The problem here is that a test run went from taking about 5 minutes to
> > taking about 17 minutes. I can reduce this to closer to 15 minutes with
> > LIBGUESTFS_BACKEND=direct and using libguestfs-make-fixed-appliance to
> > make an appliance we reuse. But that's still too long to be usable.
> > I'm hoping someone has some ideas here on how to improve things.
>
> If libguestfs is falling back to slow emulation because /dev/kvm isn't
> available, maybe it's appropriate to check for that and skip the fs tests...
This is, I think, part of the problem. We're jumping through a huge
number of hoops to avoid "sudo mount ..." in the tests directly. I
thought I was getting kvm used locally, but nope, it's not. I'm going
to take another break and see if I can figure out what I'm doing wrong
since:
sudo docker run --privileged --cap-add SYS_ADMIN --security-opt
apparmor=unconfined --rm -v /dev/fuse:/dev/fuse -v /dev/kvm:/dev/kvm
isn't doing it.
--
Tom
signature.asc
Description: PGP signature
