Hi Thomas! On Sat, Jul 13, 2019 at 06:21:30PM +0200, Thomas Schmitt wrote: > >this text is about the partition table layouts of EFI-ready bootable ISO 9660 >images and the opportunity to unify and improve them. > >Overview: >- Situation >- Flaws and remedy options >- Change proposals for debian-cd >- Change proposals for live-wrapper >- Examples of ISO partition layouts and El Torito
Thanks for your thorough work here! >------------------------------------------------------------------------ >Situation: > >UEFI specifies that on hard-disk-like media a GPT or a MBR partition >table leads to the block range of a FAT filesystem with start-up programs. >The same block range is marked by the El Torito catalog for booting from >CD, DVD, or BD media. > >There are mainly three partition table layouts of ISO 9660 images around >for this purpose: > >- Fully specs compliant MBR partition table with partition of type 0xef > and a partition for mounting the ISO 9660 filesystem. > Produced by debian-cd for arm64 and ia64. > >- Barely specs compliant MBR partition table with nested partitions > and invalid GPT. The outer partition is mountable as ISO 9660 filesystem, > the inner one is of type 0xef (and mountable as FAT). > Invented for program isohybrid option -u. > Produced by debian-cd for i386 and amd64. > >- Fully specs compliant GPT announced by Protective MBR. > Produced by grub-mkrescue. > >For tangible info about partitions and El Torito of existing Debian ISOs >see below "Examples". > >------------------------------------------------------------------------ >Flaws and remedy options: > >GPT partitioning of an image file has the disadvantage that the GPT backup >at the end of the image will not sit at the end of the physical storage >device after the ISO was copied to it. >This causes complaints from Linux when assessing the partition table and >demands the expert operation of GPT backup moving before a partition editor >can create data partitions for the unised space on the storage device. Right. >MBR partitioning with nested partitions is disallowed by UEFI. Only the >hack of setting the type of the outer partition to 0x00 prevents EFI >from recognizing the bad situation. Only by luck, Linux still recognizes >that partition as valid device. >Partition editors either complain or ignore the outer partiton. >The presence of unannounced GPT and backup GPT data makes them believe >that a clueless tool has mistaken a GPT Protective MBR for the place >where to add more partitions. ACK. >So really useful for USB sticks is only a fully specs compliant MBR >partition table. Intermediate congrats to arm64 and ia64. > >But the Debian ISOs for arm64 and ia64 have two disadvantages towards >the ISOs for amd64: > >- The ISO 9660 superblock cannot claim the blocks of the EFI partition > as part of its realm. That's because ISO 9660 partition 1 starts at > LBA 0 and does not include the EFI partition. > This causes program /sbin/isosize to report a size smaller than the > ISO image file. > This is bad for verifying a Debian ISO that is already copied to USB > stick or a DVD type which returns more blocks than were written. > xorriso-1.5.0 can prevent this if partition start is not LBA 0. > >- The EFI partition image is contained twice in the ISO image. Once as > data file and referred to by El Torito. Once as appended partition of > type 0xef. > The reason is that prior to xorriso-1.4.6 it was not possible to let > El Torito point to a block range that is not a data file in the ISO. > >xorriso-1.5.0 is now in the stable release. So it seems time to tackle >both problems: Well, ish. It might be in Buster (Debian 10.x), but our build machine casulana is still using Stretch (Debian 9.x) for now - it hasn't yet been upgraded. Once that's happened I'll look through your proposed changes and do some testing. For now we're using xorriso 1.4.6. >- Quite old xorrisofs option > -partition_offset 16 > moves the start of the ISO 9660 partition to 512-byte block 64 and thus > gives xorriso the option to claim the full image size in the ISO > superblock (PVD) at offset 0. > The prize is a second superblock at 2048-block offset 16 and a second > directory tree which will be used when the partition gets mounted. > This superblock claims only the partition's range as its realm. > > xorriso-1.5.0 takes advantage of the situation and writes the full image > file size into the superblock at offset 0, which is in no partition. > /sbin/isosize reads that superblock and thus tells the full image size. > >- Since xorriso-1.4.6 it is possible to let El Torito point to an > appended EFI partition by using > -e --interval:appended_partition_2:all:: > instead of > -e boot/grub/efi.img > Therefore no copy of the EFI partition image is needed as data file > inside the image. > This reduction of storage space can compensate for the duplication of > superblock and directory tree in the proposal above. OK, cool. >------------------------------------------------------------------------ >Change proposals for debian-cd: > >So i propose to append for amd64 and i386 the EFI partition as it is >done for arm64 and ia64. > >- Remove xorrisofs options > > -isohybrid-gpt-basdat -isohybrid-apm-hfsplus > >- Add xorrisofs option > > -append_partition 2 0xef CD1/boot/grub/efi.img > >The next two proposals apply to all four EFI bootable architectures. > >- Add xorrisofs option > > -partition_offset 16 > >- Move boot/grub/efi.img out of ./CD1 and append that file instead of > the above -append_partition gesture: > Then let El Torito point to the appended partition: > > -append_partition 2 0xef ...somewhere.else.../efi.img > > -e --interval:appended_partition_2:all:: > > (instead of: -e boot/grub/efi.img) > >Plus some cosmetic change, possible since xorriso-1.4.8: > >- Add option to set the type of partition 1 to 0x83 "Linux" (default is 0xcd) > > -iso_mbr_part_type 0x83 > >------------------------------------------------------------------------ >Change proposals for live-wrapper: > >These are the native xorriso command equivalents of above proposals. > >- Remove xorriso command > > -boot_image isolinux partition_entry=gpt_basdat > >- Add xorriso commands > > -append_partition 2 0xef ...somewhere.outside.input.tree.../efi.img > > -boot_image any partition_offset=16 > >- Change xorriso command (from "efi_path=boot/grub/efi.img") > > -boot_image any efi_path=--interval:appended_partition_2:all:: > >The input tree in debian-live-9.8.0-amd64-xfce.iso was >/w/work/free/xfce/tmp/tmpBXqfAP . As with debian-cd, efi.img should not >be in there, because it is anyways in partition 2. > >- Add xorriso command > > -boot_image any iso_mbr_part_type=0x83 > >======================================================================== >Examples of ISO partition layouts and El Torito: > >This final part of my text is of interest only if there are questions what >i am writing about. The reader may skip it if not interested in the boot >sector and partition table details of existing and proposed ISOs. >In this case: Thank you for reading up to here. > > >The following reports were obtained by > > xorriso -indev "$ISO" -report_system_area plain -report_el_torito plain > >------------------------------------------------------------------------ >Debian amd64 as is now: >------------------------------------------------------------------------ >Volume id : 'Debian 9.3.0 amd64 n' >System area options: 0x00000102 >System area summary: MBR isohybrid cyl-align-on GPT APM >ISO image size/512 : 593920 >Partition offset : 0 >MBR heads per cyl : 64 >MBR secs per head : 32 >MBR partition table: N Status Type Start Blocks >MBR partition : 1 0x80 0x00 0 593920 >MBR partition : 2 0x00 0xef 3760 832 >MBR partition path : 2 /boot/grub/efi.img >GPT : N Info >GPT disk GUID : 6215a6529f54bb43805f66f9583c139f >GPT entry array : 12 208 overlapping >GPT lba range : 64 593866 593919 >GPT partition name : 1 490053004f00480079006200720069006400 >GPT partname local : 1 ISOHybrid >GPT partition GUID : 1 6215a6529f54bb43805e66f9583c139f >GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7 >GPT partition flags: 1 0x1000000000000001 >GPT start and size : 1 0 593864 >GPT partition name : 2 490053004f004800790062007200690064003100 >GPT partname local : 2 ISOHybrid1 >GPT partition GUID : 2 6215a6529f54bb43805d66f9583c139f >GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7 >GPT partition flags: 2 0x1000000000000001 >GPT start and size : 2 3760 832 >GPT partition path : 2 /boot/grub/efi.img >APM : N Info >APM block size : 2048 >APM gap fillers : 0 >APM partition name : 1 EFI >APM partition type : 1 Apple_HFS >APM start and size : 1 940 208 >APM partition path : 1 /boot/grub/efi.img >El Torito catalog : 939 1 >El Torito cat path : /isolinux/boot.cat >El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA >El Torito boot img : 1 BIOS y none 0x0000 0x00 4 1148 >El Torito boot img : 2 UEFI y none 0x0000 0x00 832 940 >El Torito img path : 1 /isolinux/isolinux.bin >El Torito img opts : 1 boot-info-table isohybrid-suitable >El Torito img path : 2 /boot/grub/efi.img >------------------------------------------------------------------------ >Note the absence of partition 0xee in MBR and the presence of 0xef. >So the GPT info is not properly announced. >The Apple Partition Map is useless anyways, because it points to no HFS+ >filesystem. > >------------------------------------------------------------------------ >Debian arm64 as is now: >------------------------------------------------------------------------ >Volume id : 'Debian 9.4.0 arm64 n' >System area options: 0x00000b00 >System area summary: MBR cyl-align-all >ISO image size/512 : 411648 >Partition offset : 0 >MBR heads per cyl : 64 >MBR secs per head : 32 >MBR partition table: N Status Type Start Blocks >MBR partition : 1 0x00 0x83 0 411648 >MBR partition : 2 0x00 0xef 411648 2048 >El Torito catalog : 907 1 >El Torito cat path : /boot.catalog >El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA >El Torito boot img : 1 UEFI y none 0x0000 0x00 832 908 >El Torito img path : 1 /boot/grub/efi.img >------------------------------------------------------------------------ > >------------------------------------------------------------------------ >grub-mkrescue with packages "grub-pc", "grub-efi-ia32-bin", and >"grub-efi-amd64-bin": >------------------------------------------------------------------------ >Volume id : 'ISOIMAGE' >System area options: 0x00004201 >System area summary: MBR protective-msdos-label grub2-mbr cyl-align-off GPT APM >ISO image size/512 : 34780 >Partition offset : 0 >MBR heads per cyl : 64 >MBR secs per head : 32 >MBR partition table: N Status Type Start Blocks >MBR partition : 1 0x00 0xee 1 34779 >GPT : N Info >GPT disk GUID : faf27d34cdf44449a198b767638986bd >GPT entry array : 20 176 separated >GPT lba range : 64 34734 34779 >GPT partition name : 1 4700610070003000 >GPT partname local : 1 Gap0 >GPT partition GUID : 1 faf27d34cdf44449a199b767638986bd >GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7 >GPT partition flags: 1 0x1000000000000001 >GPT start and size : 1 64 276 >GPT partition name : 2 >450046004900200062006f006f007400200070006100720074006900740069006f006e00 >GPT partname local : 2 EFI boot partition >GPT partition GUID : 2 faf27d34cdf44449a19ab767638986bd >GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b >GPT partition flags: 2 0x1000000000000001 >GPT start and size : 2 340 5760 >GPT partition path : 2 /efi.img >GPT partition name : 3 48004600530050004c0055005300 >GPT partname local : 3 HFSPLUS >GPT partition GUID : 3 faf27d34cdf44449a19bb767638986bd >GPT type GUID : 3 005346480000aa11aa1100306543ecac >GPT partition flags: 3 0x1000000000000001 >GPT start and size : 3 6100 28032 >GPT partition name : 4 4700610070003100 >GPT partname local : 4 Gap1 >GPT partition GUID : 4 faf27d34cdf44449a19cb767638986bd >GPT type GUID : 4 a2a0d0ebe5b9334487c068b6b72699c7 >GPT partition flags: 4 0x1000000000000001 >GPT start and size : 4 34132 600 >APM : N Info >APM block size : 2048 >APM gap fillers : 2 >APM partition name : 1 Gap0 >APM partition type : 1 ISO9660_data >APM start and size : 1 16 1509 >APM partition name : 2 HFSPLUS_Hybrid >APM partition type : 2 Apple_HFS >APM start and size : 2 1525 7008 >APM partition name : 3 Gap1 >APM partition type : 3 ISO9660_data >APM start and size : 3 8533 162 >El Torito catalog : 1672 1 >El Torito cat path : /boot.catalog >El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA >El Torito boot img : 1 BIOS y none 0x0000 0x00 4 4047 >El Torito boot img : 2 UEFI y none 0x0000 0x00 5760 85 >El Torito img path : 1 /boot/grub/i386-pc/eltorito.img >El Torito img opts : 1 boot-info-table grub2-boot-info >El Torito img path : 2 /efi.img >------------------------------------------------------------------------ >Here we have MBR partition 0xee announcing GPT. >The Apple Partition Map points to the HFS+ superblock, directory tree, >and the ISO's data file content blocks. The APM partition is probably >not mountable by Linux because of its block size 2048. >Thus the ISO is mountable on USB by Linux only as base device, like >/dev/sdc. Not from any /dev/sdcN. > > >Now for my proposals. > >------------------------------------------------------------------------ >Debian arm64 as proposed for arm64 and ia64: >------------------------------------------------------------------------ >Volume id : 'Debian 9.4.0 arm64 n' >System area options: 0x00000a00 >System area summary: MBR cyl-align-off >ISO image size/512 : 414228 >Partition offset : 16 >MBR heads per cyl : 64 >MBR secs per head : 32 >MBR partition table: N Status Type Start Blocks >MBR partition : 1 0x00 0x83 64 413332 >MBR partition : 2 0x00 0xef 413396 832 >El Torito catalog : 1808 1 >El Torito cat path : /boot.catalog >El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA >El Torito boot img : 1 UEFI y none 0x0000 0x00 832 103349 >El Torito img blks : 1 208 >------------------------------------------------------------------------ > >------------------------------------------------------------------------ >Debian amd64 as proposed for amd64 and i386 (with BIOS boot stuff): >------------------------------------------------------------------------ >Volume id : 'Debian 9.3.0 amd64 n' >System area options: 0x00000102 >System area summary: MBR isohybrid cyl-align-on >ISO image size/512 : 691008 >Partition offset : 16 >MBR heads per cyl : 64 >MBR secs per head : 32 >MBR partition table: N Status Type Start Blocks >MBR partition : 1 0x80 0x83 64 690112 >MBR partition : 2 0x00 0xef 690176 832 >El Torito catalog : 1872 1 >El Torito cat path : /isolinux/boot.cat >El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA >El Torito boot img : 1 BIOS y none 0x0000 0x00 4 1873 >El Torito boot img : 2 UEFI y none 0x0000 0x00 832 172544 >El Torito img path : 1 /isolinux/isolinux.bin >El Torito img opts : 1 boot-info-table isohybrid-suitable >El Torito img blks : 2 208 >------------------------------------------------------------------------ > >(My repacked ISOs are slightly larger than originals, because mounting > -t iso9660 regrettably does not show the hardlink relations in the tree.) > > >Have a nice day :) > >Thomas > > -- Steve McIntyre, Cambridge, UK. st...@einval.com Who needs computer imagery when you've got Brian Blessed?
