On Fri, 11 Dec 2009, Joey Hess wrote:
BTW, if you enjoy the detective work of tracking these down, check out the 'testsuite' branch of pristine-tar's git repo for the full collection of files it is not yet able to reproduce w/o deltas.
First I went for the low-hanging fruit in pristine-tar-2e93c8baafb3d075a2bdf592c3deb7356fbfda37, namely "gzip -t -v *.gz". gzip handled 140 of 149 files with a simple OK, the remaining 9 files can be categorized like this:
a) "trailing zero bytes ignored": 11e1e4ce71e8223ff1b40855c3adef899255eb50 ndoc_1.3.1.orig.tar.gz 8616ed97e7dc2e67319abfbc26a49fa9cf2c9f00 resample_1.7.orig.tar.gz 893bd3ddfe59a444c577b3576abe63818244ca86 wapiti_1.1.6.orig.tar.gz 01443bd19f640708f72038e57dfdd8ee3ee0af21 xsupplicant_1.2.4.dfsg.1.orig.tar.gz b) "extra field of 261 bytes ignored": 877ea5b283060fe0160e376ea645e8e168047ff5 openssh_5.1p1.orig.tar.gz 966f23493e372728f817d95b5879705e21572ed9 py-radix_0.5.orig.tar.gz 33f96e774e086e15a52893e18aa56ed64f3fe632 python-bcrypt_0.1.orig.tar.gz c) "extra field of 1029 bytes ignored": 2064cad001b742b70fd2f44fdcab02c4fcb6c18d libbsd-arc4random-perl_1.3.orig.tar.gz 126b18c1b90c99f2518dbc67d2287d17824dd160 mksh_36.2.orig.tar.gzI have modified gzip to dump the skipped extra fields. In both affected categories, that is, (b) and (c), each file has the field starting with \x47\x53, denoting (according to the to-be-fixed description under [0]) a "GS" (gzsig) extra field. (Googling for gzsig makes it appear wide-spread on *BSD.)
Then I've remembered Paul Sladen's invaluable "pyflate" utility [1], which I had discovered first when I started writing lbzip2. I've re-enabled some commented out debug messages in the gzip_main() method, changed a _read() method call to read(), and commented out some other messages. (See attached file "pyflate-patch".)
I have recompressed all of the 149 files in a separate directory with gzip --best. Although file modification times were preserved during recompression, I suppose the git->tar export has previously lost the original mtimes when I've downloaded pristine-tar, because now each such file has mtime 2009-12-11 21:28:58 CET on my disk.
I have run the modified pyflate on both sets of files (original and recompressed) and compared the debug output for each pair of files.
* In the original files, the only "extra_flags" value is 0x0. In the recompressed files, the only "extra_flags" value is 0x2, meaning "maximum compression"; see 2.3.1./XFL in [2]. I believe this follows from me passing --best to gzip.
* In the original files, "flags" values cover the set { 0x0, 0x4 }. 0x4 means FEXTRA (extra fields present), see 2.3.1./FLG in [2]. The affected files are those mentioned above in categories (b) and (c). The only "flags" value occurring in the recompressed files is 0x8, FNAME ("an original file name is present"). None of the 149 original files have the FNAME flag set.
* There are different "mtime" headers in the original files. Notably, the mtime value 0x0 occurs ("no time stamp is available"; see 2.3.1./MTIME in [2].) The recompressed files all have mtime 0x4b22ab8a, corresponding to the time stamp mentioned above:
$ echo $(( 16#4b22ab8a - $(date -d '2009-12-11 21:28:58 CET' +%s) )) 0* In the original files, the "os_type" values cover the following set (see [0]):
0x00 - FAT file system (DOS, OS/2, NT) + PKZIPW 2.50 VFAT, NTFS; 0x06 - HPFS file system (OS/2, NT 3.x); 0x07 - Macintosh; 0x0b - NTFS file system (NT); 0x0e - VFAT file system (Win95, NT); 0xff - <invalid>.Most notably, 0x03 (Unix) is missing; no original file was compressed on such an OS. Obviously, the recompressed files all have 0x03 in their "os_type" headers.
* These findings are probably not exhaustive. I've attached a compressed archive of all pyflate-log files. Cheers, lacos [0] http://www.gzip.org/format.txt [1] http://www.paul.sladen.org/projects/compression/ [2] http://www.ietf.org/rfc/rfc1952.txt
pyflate-logs.tar.bz2
Description: Binary data
--- /home/lacos/tmp/pyflate-0.31+bzr20070122/pyflate.py.orig 2007-01-22
02:31:41.000000000 +0100
+++ /home/lacos/tmp/pyflate-0.31+bzr20070122/pyflate.py 2009-12-12
01:52:30.000000000 +0100
@@ -45,7 +45,7 @@
while n >= self.bits and n > 7:
n -= self.bits
self.bits = 0
- n -= len(self.f._read(n >> 3)) << 3
+ n -= len(self.f.read(n >> 3)) << 3
if n:
self.readbits(n)
# No return value
@@ -451,13 +451,13 @@
# Use flags, drop modification time, extra flags and OS creator type.
flags = b.readbits(8)
- #print 'flags', hex(flags)
+ print 'flags', hex(flags)
mtime = b.readbits(32)
- #print 'mtime', hex(mtime)
+ print 'mtime', hex(mtime)
extra_flags = b.readbits(8)
- #print 'extra_flags', hex(extra_flags)
+ print 'extra_flags', hex(extra_flags)
os_type = b.readbits(8)
- #print 'os_type', hex(os_type)
+ print 'os_type', hex(os_type)
if flags & 0x04: # structured GZ_FEXTRA miscellaneous data
xlen = b.readbits(16)
@@ -471,10 +471,10 @@
if flags & 0x02: # header-only GZ_FHCRC checksum
b.readbits(16)
- #print "gzip header skip", b.tell()
+ print "gzip header skip", b.tell()
out = ''
- #print 'header 0 count 0 bits', b.tellbits()
+ print 'header 0 count 0 bits', b.tellbits()
while True:
header_start = b.tell()
@@ -483,7 +483,7 @@
lastbit = b.readbits(1)
#print "last bit", hex(lastbit)
blocktype = b.readbits(2)
- print "deflate-blocktype", blocktype, 'beginning at', header_start
+ #print "deflate-blocktype", blocktype, 'beginning at', header_start
#print 'raw block data at', b.tell()
if blocktype == 0:
@@ -615,11 +615,11 @@
b.align()
crc = b.readbits(32)
final_length = b.readbits(32)
- #print len(out)
+ print len(out)
next_unused = b.tell()
#print 'deflate-end-of-stream', 5, 'beginning at', footer_start, 'raw data
at', next_unused, 'bits', b.tellbits() - bfooter_start
print 'deflate-end-of-stream'
- #print 'crc', hex(crc), 'final length', final_length
+ print 'crc', hex(crc), 'final length', final_length
#print 'header 0 count 0 bits', b.tellbits()-bfooter_start
return out
