This bug bit me in packaging libcec. Lintian says
unstripped-static-library, which dh_strip should be stripping but
doesn't.
I added a stanza to libcec debian/rules to strip the file explicitly
instead of relying on dh_strip, but that's annoying.
Had claude look at the issue. It independently found the present bug
in dh_strip. And then it found this particular bug report. But it also
(a) did a deeper analysis of the bug, and (b) suggested a simpler fix,
namely having the perl code for dh_strip directly check the file magic
number before switching over to heuristics.
For your AI condescension pleasure I'm including its long
claude-speak discourse on the matter below. This includes a
deeper analysis of the bug and a tiny patch.
Seriously: despite claude acting like a know-it-all we'd all like to
strangle, it seemed to nail the diagnosis and a proper fix. And this
dh_strip bug is stupid and annoying and causing technical debt
elsewhere in the project so really should be fixed.
Cheers,
--Barak.
---------------------------------------
-- BEGIN CLAUDE SPEAK --
---------------------------------------
[This is a follow-up to the existing #875780 rather than a new report:
that bug has had the symptom since 2017 but, as far as I can see from
the log, never the mechanism. Below is the mechanism, a one-line test
for it, and a patch that needs neither file(1) nor libmagic, which I
think answers the objections that stalled the earlier attempts.]
Package: debhelper
Version: 14.3
Severity: normal
I hit this in src:libcec, where libcec-dev shipped a 22MB libcec.a with
full debug info. lintian says unstripped-static-library; dh_strip issues
no strip command for the file at all.
The cause is the `-B` guard in dh_strip's testfile():
# Is it a static library, and not a debug library?
if ($fn =~ m/\/lib[^\/]*\.a$/ && $fn !~ m/.*_g\.a$/) {
# Is it a binary file, or something else (maybe a linker
# script on Hurd, for example? I don't use file, because
# file returns a variety of things on static libraries.
if (-B $fn) {
push @static_libs, $fn;
return;
}
}
The filename regex matches. `-B` is what fails.
WHY -B FAILS
Earlier messages in this bug describe perl's -T/-B as a statistical
heuristic over "odd" characters, and reason about whether an archive has
enough of them. That is only the second half of the rule. perlfunc:
The -T and -B tests work as follows. The first block or so of the
file is examined to see if it is valid UTF-8 that includes non-ASCII
characters. If so, it's a -T file. Otherwise, that same portion of
the file is examined for odd characters such as strange control
codes or characters with the high bit set. If more than a third of
the characters are strange, it's a -B file; otherwise it's a -T file.
So the odd-character count is only reached if the UTF-8 test fails.
It is the UTF-8 branch that fires here, and it fires on data that is
overwhelmingly "odd":
$ perl -e 'printf "size=%d -B=%d -T=%d\n", -s $ARGV[0],
(-B $ARGV[0])?1:0, (-T $ARGV[0])?1:0' libcec.a
size=22656606 -B=0 -T=1
443 of the first 512 bytes are high-bit or control characters, i.e.
the fallback rule would have said "binary" with room to spare.
The reason the block is valid UTF-8 is the ar symbol table. Its entries
are 4-byte big-endian member offsets, and libcec.a's first member sits
at 0x0001CD8E, so the table is this, over and over:
0000100 ` \n \0 \0 \b \0 \0 001 315 216 \0 001 315 216
0000120 \0 001 315 216 \0 001 315 216 \0 001 315 216 \0 001 315 216
*
0xCD 0x8E is a well-formed two-byte sequence (U+034E), 0x00 and 0x01 are
ASCII, and the ar header preceding it is all ASCII. The whole 512-byte
block therefore decodes:
$ perl -MEncode -e 'open(F,"<",$ARGV[0]); binmode F; read(F,$b,512);
print eval { Encode::decode("UTF-8",$b,Encode::FB_CROAK); 1 }
? "valid UTF-8\n" : "not UTF-8\n"' libcec.a
valid UTF-8
To confirm that this branch, and not the odd-character count, is what
decides the outcome, take the same block and corrupt one continuation
byte into an invalid lead byte. The high-bit count is unchanged; only
UTF-8 validity changes:
A (block as-is) odd=443/512 -B=0 -T=1
B (one byte -> 0xC0) odd=443/512 -B=1 -T=0
(perlfunc separately says a file with a zero byte in the examined
portion is considered binary. That is not what the implementation does:
the block above is full of NULs and still comes out -T. The UTF-8 test
wins. Worth knowing before assuming a NUL-check would be enough here.)
So this is not "archives are marginally texty and sometimes tip over the
1/3 threshold". It is a hard switch: an archive is misclassified exactly
when its symbol table's member offsets happen to land in byte ranges
that form valid UTF-8. That is a function of member sizes, which is why
Ximin Luo's original report saw libcore.rlib and libstd.rlib disagree,
why MariaDB's libraries "changed from ArFile to StaticLibFile" between
build dates, and why the bug looks nondeterministic. It is perfectly
deterministic; it just depends on data no one controls deliberately.
PROPOSED FIX
ar archives have an 8-byte magic. Reading it is cheaper than -B (which
reads a whole block and scans it), needs no subprocess, and adds no
dependency -- so it sidesteps both Niels' objection to file(1) and the
libfile-libmagic-perl suggestion. The helper below is deliberately
written in the same shape as the existing is_so_or_exec_elf_file(), and
gives the guard the semantics its comment always claimed.
It also keeps the case that guard exists for: a Hurd linker script named
libfoo.a has no ar magic and is still skipped.
--- a/lib/Debian/Debhelper/Dh_Lib.pm
+++ b/lib/Debian/Debhelper/Dh_Lib.pm
@@ -151,6 +151,7 @@
rm_files
excludefile
is_so_or_exec_elf_file
+ is_ar_archive
is_empty_dir
reset_perm_and_owner
log_installed_files
@@ -3161,6 +3162,23 @@
ELF_TYPE_SHARED_OBJECT => 0x0003,
};
+use constant AR_MAGIC => "!<arch>\n";
+
+sub is_ar_archive {
+ my ($file) = @_;
+ open(my $fd, '<:raw', $file) or error("open $file: $!");
+ my $buflen = 0;
+ my $buf;
+ while ($buflen < length(AR_MAGIC)) {
+ my $r = read($fd, $buf, length(AR_MAGIC) - $buflen, $buflen)
// error("read ($file): $!");
+ last if $r == 0; # EOF
+ $buflen += $r;
+ }
+ close($fd);
+ return 0 if $buflen < length(AR_MAGIC);
+ return $buf eq AR_MAGIC ? 1 : 0;
+}
+
sub is_so_or_exec_elf_file {
my ($file) = @_;
open(my $fd, '<:raw', $file) or error("open $file: $!");
--- a/dh_strip
+++ b/dh_strip
@@ -239,10 +239,9 @@
}
# Is it a static library, and not a debug library?
if ($fn =~ m/\/lib[^\/]*\.a$/ && $fn !~ m/.*_g\.a$/) {
- # Is it a binary file, or something else (maybe a linker
- # script on Hurd, for example? I don't use file, because
- # file returns a variety of things on static libraries.
- if (-B $fn) {
+ # Is it really an ar archive, or something else (maybe a
+ # linker script on Hurd, for example)?
+ if (is_ar_archive($fn)) {
push @static_libs, $fn;
return;
}
TESTING
Package tree containing the real libcec.a plus a fake Hurd-style linker
script named libscript.a. Same tree, both runs, debhelper 14.3,
perl 5.42.2.
stock dh_strip -v:
(no strip command issued)
libcec.a 22656606 -> 22656606
libscript.a 65 -> 65
patched dh_strip -v:
strip --strip-debug --remove-section=.comment --remove-section=.note
--enable-deterministic-archives -R .gnu.lto_* -R .gnu.debuglto_*
-N __gnu_lto_slim -N __gnu_lto_v1 .../libcec.a
libcec.a 22656606 -> 2315086
libscript.a 65 -> 65
So the archive gets stripped, the linker script is still left alone, and
lintian's unstripped-static-library goes away. In the real package this
took libcec-dev from 2.9MB to 330kB.
NOTE ON SCOPE
The same misclassification will apply to anything else gated on -B. I
have only verified the static library path. Rust .rlib files are also ar
archives and were mentioned upstream in this bug, but they do not match
dh_strip's /lib[^\/]*\.a$/ pattern, so whatever affects them is a
separate question from this one.
