Porting the standard image from live-wrapper to live-build

2020-11-11 Thread Roland Clobus
Hello Debian-Live list and Debian-Images list,

On 2020-03-21T17:27+0200 [1] I wrote these mailing lists about the
generation of the official live images for Debian. Since then I have
announced my effort to make the generation of live images reproducible
[2] and have come really far (for the standard image). As detailed in my
blog [3] I can re-generate the official Debian 10.6 Live Standard image
now, using live-build. Since vmdebootstrap isn't available in the
current Debian Testing, and appears not to be planned to be included,
I've prepared an image that is as similar as feasible/meaningful using
the latest git version of live-build (with my own patches for adding
reproducibilty and a few hacks on top) [4].
With Debian point release 10.7 announced [5], now would be perhaps an
ideal moment to make the switch to live build.

This is a long mail with many questions, but please take the time to
answer some...

First, my restrictions/goals:
* While testing, use as little bandwidth as possible (therefore using
apt-cacher-ng)
* Re-create the live-wrapper image for Debian 10.6 in the standard
configuration as close as possible
* Try to make a reasonable well reproducible live image
* Have reasonable short test cycles (therefore using /dev/shm)

Below follows a list of differences between the live-wrapper and
live-build images, and my personal opinion about them. Some features are
not present in live build and needs further action (marked with AP).
Please answer to this mail to confirm or reject my proposals:

live-wrapper:
* ISO volume ID contains '10.6.0'
** AP: live build mentions 'buster', which contains less details
* Makes a beep on boot
** AP: Should this be added to live build as well?
* The grub configuration contains 'SAYS ...'
** AP: Should this be added to live build as well?
* /pool contains some packages that are not in the squashfs image
** live build has a more minimal list
* /pool contains exim4, mailutils, mokutil, python2.7, but it is not in
the squashfs
** live build has a more minimal list
* /pool contains multiple versions of udeb files
** live build has only the active version
* The grub and isolinux menus contain all available languages
** AP: It would be nice to reproduce this in live build
* Contains /etc/hosts, /etc/machine-id, several logs in /var/log,
references to contrib and non-free in /var/lib/apt/lists
** live build cleans up these files better
* Does not contain apparmor (recommends of linux-image-4.19.0-11-amd64)
** live build contains all recommends packages
* Does not contain acpi-support-base (recommends of acpid)
** live build contains all recommends packages
* Contains /etc/modprobe.d/qemu-blacklist.conf (for bochs-drm)
** AP: Is this also needed in live build?
* Encoding is us-ascii, live build uses utf-8
** I think utf-8 would be the best
* The boot splash screen uses the Debian theme
** AP: live build shows a helmet and the versions of the live packages.
We need the Debian-themed splash screen, combined with the version numbers

live-build:
* /EFI/boot contains a 32-bit EFI image on the amd64 iso.
** AP: Is this needed/correct?
* /pool contains firmware-free
** This is missing in the live-wrapper image
* boot/grub/grub.cfg: No findiso= line in the fail-safe mode
** AP: Untested by me, is it a bug?
* xorriso complains about issues with Joliet (symlinks not supported,
volid too long)
** AP: Do we need support for Joliet? Untested: does Windows XP/7/10
support RockRidge sufficiently well? And is it needed?
* The packages lists are available both uncompressed and as gzip file
** AP: Isn't just one variant suffient? live-wrapper only has the
uncompressed file

My comments to the command line options to lb config (as shown below):
* --security false
** AP: Shouldn't this be true per default for Debian Stable?
* --updates false
** AP: Shouldn't this be true per default for Debian Stable?
* --loadlin false
** Is this still tested? (I don't have a computer which can run 16-bit
executables at the moment)
* acpid
** If this is needed, shouldn't if be in the live-task-standard package?
* Rename install to d-i
** Only needed to make the image more similar to the live-wrapper image
-> should not be merged to the git repository of live-build
* The git repo [4] has 2 commits which start with 'HACK'
** Only needed to make the image more similar to the live-wrapper image
-> should not be merged to the git repository of live-build

With kind regards,
Roland Clobus

--- Appendix: Additional comparison on top of the result of diffoscope ---
# Compare the squashfs images
# 1. Align the timestamps to the official image. Everything after 10:00
will get the SOURCE_DATE_EPOCH timestamp
# 2. Remove the directories from the diff -> they might have different
'sizes', which are not of interest
TZ=UTC unsquashfs -lls live-wrapper-mounted/live/filesystem.squashfs |
sed -e "s/2020-09-26 10:[0-9][0-9]/2020-09-26 10:42/" | sed -e "/^d/d" >
live-wrapper.squash
TZ=UTC unsquashfs -lls live-build-mounted/li

Re: Porting the standard image from live-wrapper to live-build

2020-11-11 Thread Thomas Schmitt
Hi,

Roland Clobus wrote:
> * xorriso complains about issues with Joliet (symlinks not supported,
> volid too long)

Joliet uses 16-bit characters which halves the field length in the
supplementary volume descriptor of Joliet. So no workaround is possible
other than shortening the volume id.

If there is knowledge how to represent symbolic links in Joliet so that
MS-Windows understands them, i'd be willing to work on an implementation
in libisofs.


> ** AP: Do we need support for Joliet? Untested: does Windows XP/7/10
> support RockRidge sufficiently well? And is it needed?

I am not aware that any MS-Windows system supports Rock Ridge natively.
There might be add-on software which can.

Having Joliet filenames of up to 64 characters length is better than
having ISO 9660 names of limited charset and max length 31.
It opens a better path to a copy method onto USB stick which is popular
among MS users. They simply copy all files from the ISO into a FAT
partition.
If a copy of /EFI/BOOT from the ISO's EFI partition is among them or gets
added, then the result is said to boot on all EFI firmwares.

See the complaint of the author of Rufus when Ubuntu hid the /EFI/BOOT
equipment in its new ISO layout:
  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895131

When looking into mounted
  debian-live-10.2.0-amd64-xfce+nonfree.iso
  debian-10.6.0-amd64-netinst.iso
i see
  /EFI/boot
  /EFI/boot/bootx64.efi
  /EFI/boot/grubx64.efi
The files match those in the EFI partition image
  /boot/grub/efi.img
But the partition image contains an additional file
  /efi/debian/grub.cfg

Maybe one should put it into the /EFI/boot/ directory of the ISO ?


Have a nice day :)

Thomas