Re: Same hashfiles for files that should be different from each other?
Hi, (Cc-ing debian-l...@lists.debian.org) Evert Kuijpers wrote to debian-cd: > Both debian-live-9.8.0-i386-lxde.contents and > debian-live-9.8.0-i386-mate.contents have all the same hashes. > Both debian-live-9.8.0-amd64-lxde.contents and > debian-live-9.8.0-amd64-mate.contents have all the same hashes. > It is quite possible that two of these four files should have different > contents. > > See https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/ > and https://cdimage.debian.org/debian-cd/current-live/i386/iso-hybrid/ > > Greetings from Evert Kuijpers, Tilburg in The Netherlands, hyss...@gmail.com That's because they have identical file content, because the trees of the ISOs bear identical file paths which nearly match the .content paths list. (The file paths "/isolinux/boot.cat" are not in .content, because these file paths got created by option -c of xorrisofs when the ISO was produced.) So the question is whether it is normal that both ISOs have absolutely identical file paths in their trees. The same checksum duplicity can be seen with 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-cinnamon.contents 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-gnome.contents Have a nice day :) Thomas
Re: Same hashfiles for files that should be different from each other?
On 30/03/2019 08:49, Thomas Schmitt wrote: Hi, (Cc-ing debian-l...@lists.debian.org) Evert Kuijpers wrote to debian-cd: Both debian-live-9.8.0-i386-lxde.contents and debian-live-9.8.0-i386-mate.contents have all the same hashes. Both debian-live-9.8.0-amd64-lxde.contents and debian-live-9.8.0-amd64-mate.contents have all the same hashes. It is quite possible that two of these four files should have different contents. See https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/ and https://cdimage.debian.org/debian-cd/current-live/i386/iso-hybrid/ Greetings from Evert Kuijpers, Tilburg in The Netherlands, hyss...@gmail.com That's because they have identical file content, because the trees of the ISOs bear identical file paths which nearly match the .content paths list. (The file paths "/isolinux/boot.cat" are not in .content, because these file paths got created by option -c of xorrisofs when the ISO was produced.) So the question is whether it is normal that both ISOs have absolutely identical file paths in their trees. The same checksum duplicity can be seen with 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-cinnamon.contents 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-gnome.contents That is exactly what Kuijpers has just pointed out. Clearly something has gone wrong with the automated build somewhere - different desktop environments should have a different manifest from each other... Thank you for pointing out there is an issue. I have not looked yet - but the issue could be in one of several places. (1) The manifest is incorrect and we are building identical ISOs and just giving different names to them (2) we are building the content of the ISOs correctly, but incorrectly reporting the manifest (3) we are reporting the manifest correctly and we are reporting incorrect checksums I'll take a closer look now /Andy
Re: Same hashfiles for files that should be different from each other?
Hi, i wrote: > > The same checksum duplicity can be seen with > > 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-cinnamon.contents > > 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-gnome.contents Andy Simpkins wrote: > That is exactly what Kuijpers has just pointed out. Evert Kuijpers pointed to "lxde" and "mate" on "i386" and "amd64". My run of program "sort" reveiled that "cinnamon" and "gnome" form a pair, too (at least on "amd64"). > the issue could be in one of several places. [...] > (2) we are building the content of the ISOs > correctly, but incorrectly reporting the manifest This can be ruled out. I downloaded debian-live-9.8.0-amd64-lxde.iso and debian-live-9.8.0-amd64-mate.iso. Mounted them at /mnt/iso and compared the outputs of find /mnt/iso | sed -e 's/^\/mnt\/iso//' | sort They are identical. The results of "find" differ from the identical .content files only by the paths "/isolinux/boot.cat", for which there is a plausible explanation in the ISO production process. > (3) we are reporting the > manifest correctly and we are reporting incorrect checksums This can be ruled out. "md5sum" confirms the MD5s. "diff" confirms that the .content files have identical content. Have a nice day :) Thomas
Re: Same hashfiles for files that should be different from each other?
On 30/03/2019 09:56, Thomas Schmitt wrote: Hi, i wrote: The same checksum duplicity can be seen with 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-cinnamon.contents 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-gnome.contents Andy Simpkins wrote: That is exactly what Kuijpers has just pointed out. Evert Kuijpers pointed to "lxde" and "mate" on "i386" and "amd64". My run of program "sort" reveiled that "cinnamon" and "gnome" form a pair, too (at least on "amd64"). the issue could be in one of several places. [...] (2) we are building the content of the ISOs correctly, but incorrectly reporting the manifest This can be ruled out. I downloaded debian-live-9.8.0-amd64-lxde.iso and debian-live-9.8.0-amd64-mate.iso. Mounted them at /mnt/iso and compared the outputs of find /mnt/iso | sed -e 's/^\/mnt\/iso//' | sort They are identical. The results of "find" differ from the identical .content files only by the paths "/isolinux/boot.cat", for which there is a plausible explanation in the ISO production process. (3) we are reporting the manifest correctly and we are reporting incorrect checksums This can be ruled out. "md5sum" confirms the MD5s. "diff" confirms that the .content files have identical content. Have a nice day :) Thomas --- This email has been checked for viruses by AVG. https://www.avg.com Hi Thomas Ack. So it looks like the contents are identical. This isn't entirely surprising - the packages are after all should be the same. I am guessing at this point (and still looking) but there manifests between the different desktop environments (after the desktop environment itself) will be very similar, and only differ from one another because of the size of the desktop environment itself (i.e. a smaller environment takes up less space so there is more room for additional packages, a larger desktop has less space available so the manifest of packages is therefore smaller). Sledge would be able to confirm this with authority, but I believe he is away at the moment. I will continue digging to try and confirm this hypothesis /Andy
Re: Same hashfiles for files that should be different from each other?
On 30/03/2019 10:16, Andy Simpkins wrote: On 30/03/2019 09:56, Thomas Schmitt wrote: Hi, i wrote: The same checksum duplicity can be seen with 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-cinnamon.contents 5849124a0e25d1318a880e98b9a8123f debian-live-9.8.0-amd64-gnome.contents Andy Simpkins wrote: That is exactly what Kuijpers has just pointed out. Evert Kuijpers pointed to "lxde" and "mate" on "i386" and "amd64". My run of program "sort" reveiled that "cinnamon" and "gnome" form a pair, too (at least on "amd64"). the issue could be in one of several places. [...] (2) we are building the content of the ISOs correctly, but incorrectly reporting the manifest This can be ruled out. I downloaded debian-live-9.8.0-amd64-lxde.iso and debian-live-9.8.0-amd64-mate.iso. Mounted them at /mnt/iso and compared the outputs of find /mnt/iso | sed -e 's/^\/mnt\/iso//' | sort They are identical. The results of "find" differ from the identical .content files only by the paths "/isolinux/boot.cat", for which there is a plausible explanation in the ISO production process. (3) we are reporting the manifest correctly and we are reporting incorrect checksums This can be ruled out. "md5sum" confirms the MD5s. "diff" confirms that the .content files have identical content. Have a nice day :) Thomas --- This email has been checked for viruses by AVG. https://www.avg.com Hi Thomas Ack. So it looks like the contents are identical. This isn't entirely surprising - the packages are after all should be the same. I am guessing at this point (and still looking) but there manifests between the different desktop environments (after the desktop environment itself) will be very similar, and only differ from one another because of the size of the desktop environment itself (i.e. a smaller environment takes up less space so there is more room for additional packages, a larger desktop has less space available so the manifest of packages is therefore smaller). Sledge would be able to confirm this with authority, but I believe he is away at the moment. I will continue digging to try and confirm this hypothesis /Andy OK so whilst the .contents files are the same for debian-live-9.8.0-amd64-cinnamon.contents and debian-live-9.8.0-amd64-gnome.contents their corresponding .packages files are not the same, thus indicating (to me - perhaps mistakenly) that there is a problem and that this is NOT correct behaviour. still digging /Andy
Re: Same hashfiles for files that should be different from each other?
Hi, Evert Kuijpers wrote: > Because I question myself: How can the contents be the same, when the > associated files are different? It's only the file names and only in the ISO filesystem, which is an outer layer of the Live system's file wealth. Much more files are in the compressed squash filesystem image which is a large file in the ISO. mate: -rw-r--r-- 1 root root 1872347136 Feb 16 13:16 /live/filesystem.squashfs lxde: -rw-r--r-- 1 root root 1812656128 Feb 16 13:17 /live/filesystem.squashfs Another file storage is the initrd. It would not be astounding if that one would turn out to be the same in both ISOs. Bot the initrds differ too: mate: -rw-r--r-- 1 root root 23528076 Feb 16 13:16 /live/initrd.img-4.9.0-8-amd64 lxde: -rw-r--r-- 1 root root 23528328 Feb 16 13:17 /live/initrd.img-4.9.0-8-amd64 So for now we have no clear indication that something is wrong. Probably the system parts outside the squashfs images are similar enough to yield the same file names, if not the same file content. (I have to guess, as i am only involved in the final step of xorriso packing up the ISO from the prepared files on hard disk.) Whatever, your observation is surprising enough to deserve a thorough investigation. General comparison of the ISOs: $ diff -rq /mnt/iso /mnt/iso2 2>&1 | less yields Files /mnt/iso/.disk/info and /mnt/iso2/.disk/info differ by date and system name. Files /mnt/iso/.disk/mkisofs and /mnt/iso2/.disk/mkisofs differ by the system name in the ISO 9660 Volume Id in the xorriso command. Files /mnt/iso/dists/stretch/Release and /mnt/iso2/dists/stretch/Release differ by date. Files /mnt/iso/live/filesystem.squashfs and /mnt/iso2/live/filesystem.squashfs differ Files /mnt/iso/live/initrd.img-4.9.0-8-amd64 and /mnt/iso2/live/initrd.img-4.9.0-8-amd64 differ Both are images containing file trees. Have a nice day :) Thomas
Re: Same hashfiles for files that should be different from each other?
Hi, comparison of both squashfs images yield appeasing results: mount debian-live-9.8.0-amd64-mate.iso /mnt/iso mount /mnt/iso/live/filesystem.squashfs /mnt/squash mount debian-live-9.8.0-amd64-lxde.iso /mnt/iso2 mount /mnt/iso2/live/filesystem.squashfs /mnt/squash2 diff -rq /mnt/squash /mnt/squash2 This reports among many other differences: ... Only in /mnt/squash2/etc/xdg/menus: lxde-applications.menu Only in /mnt/squash2/etc/xdg/menus: lxlauncher-applications.menu Only in /mnt/squash/etc/xdg/menus: mate-applications.menu Only in /mnt/squash/etc/xdg/menus: mate-preferences-categories.menu ... So yes, there's "mate" in *-mate.iso and "lxde" in *-lxde.iso. Have a nice day :) Thomas
Re: Same hashfiles for files that should be different from each other?
On Sat, Mar 30, 2019 at 10:43:45AM +, Andy Simpkins wrote: >On 30/03/2019 10:16, Andy Simpkins wrote: >> >> So it looks like the contents are identical. This isn't entirely >> surprising - the packages are after all should be the same. I am >> guessing at this point (and still looking) but there manifests between >> the different desktop environments (after the desktop environment itself) >> will be very similar, and only differ from one another because of the >> size of the desktop environment itself (i.e. a smaller environment takes >> up less space so there is more room for additional packages, a larger >> desktop has less space available so the manifest of packages is therefore >> smaller). >> >> Sledge would be able to confirm this with authority, but I believe he is >> away at the moment. I will continue digging to try and confirm this >> hypothesis >> >OK so whilst the .contents files are the same for >debian-live-9.8.0-amd64-cinnamon.contents and >debian-live-9.8.0-amd64-gnome.contents their corresponding .packages files >are not the same, thus indicating (to me - perhaps mistakenly) that there is >a problem and that this is NOT correct behaviour. It's basically what Thomas has pointed out elsewhere. The contents files just list the extra files that are in the ISO. The core differences from one live image to the next are the different desktop setups, and that's all contained within the squashfs - see the *.packages files for lists of what's contained in the squashfs on each image. I'm actually slightly surprised that the contents files are not even more close together, but this is just down to dependencies. As part of the live-wrapper run, we add a list of a few extra packages that might want to be used for installation, *on top of* the desktop system in the squashfs. The exact list will be modified by dependencies: in some cases some of the needed packages will already be covered by what's in the squashfs. Does that make sense? -- Steve McIntyre, Cambridge, UK.st...@einval.com Dance like no one's watching. Encrypt like everyone is. - @torproject
