Hi Chris,
On Fri, 13 Mar 2009, Chris St Denis wrote:
I think I've found a bug in libwrap/tcpwrappers.
I think so too :) See below.
Before filing an actual bug report I want to get some feedback here
first.
A hosts.allow file with ~1000 ips on a single line (Haven't
experimented with
other quantities yet), causes network daemons that use libwrap stop
accepting
incoming network connections and use 100% cpu on an incoming connection.
This problem appeared because sshguard placed a large number of IPs
in my
hosts.allow file triggering this bug.
I've left the affected daemons for a long period of time (once about
8 hours)
and they don't seem to come back, so I think this is more than just
it taking
a while to loop through a 1000 item array of IPs
The production system that was affected is FreeBSD 7.0-32bit
Test system is FreeBSD 7.1-32bit
Example hosts.allow file (IPs are randomly generated for purposes of
example)
sshd : 112.110.123.63 113.11.2.126 113.11.8.6 113.19.19.22
113.197.48.68 <snipped 990+ IPs> 116.48.108.244 116.48.11.19 : deny
ALL : ALL : allow
top output of affected system. sshd wcpu slowly crawls up to 100%
over about
30 seconds or so.
crash# top
last pid: 692; load averages: 0.08, 0.04, 0.04
up
0+00:12:13 15:42:30
24 processes: 2 running, 22 sleeping
CPU: 49.7% user, 0.0% nice, 0.2% system, 0.2% interrupt, 49.9% idle
Mem: 9304K Active, 6004K Inact, 21M Wired, 32K Cache, 10M Buf, 947M
Free
Swap: 1995M Total, 1995M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
691 root 1 103 0 5760K 3660K CPU1 1 0:04 33.98% sshd
672 root 1 4 0 8436K 3888K sbwait 1 0:00 0.00% sshd
677 cstdenis 1 20 0 4460K 2288K pause 0 0:00 0.00% csh
682 root 1 20 0 5484K 2632K pause 0 0:00 0.00% csh
675 cstdenis 1 44 0 8436K 3896K select 0 0:00 0.00% sshd
<snip>
A backtrace shows
crash# gdb /usr/sbin/sshd 691
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...
Attaching to program: /usr/sbin/sshd, process 691
Reading symbols from /usr/lib/libssh.so.4...done.
Loaded symbols for /usr/lib/libssh.so.4
Reading symbols from /lib/libutil.so.7...done.
Loaded symbols for /lib/libutil.so.7
Reading symbols from /lib/libz.so.4...done.
Loaded symbols for /lib/libz.so.4
Reading symbols from /usr/lib/libwrap.so.5...done.
Loaded symbols for /usr/lib/libwrap.so.5
<snip other symbols for breviry>
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
/usr/src/lib/libc/stdio/fgets.c:56
56 {
(gdb) bt
#0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
/usr/src/lib/libc/stdio/fgets.c:56
#1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at
/usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38
#2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow",
request=0xbfbfeb14)
at
/usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162
#3 0x28111540 in hosts_access (request=0xbfbfeb14) at
/usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132
#4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at
/usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843
(gdb) bt
#0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
/usr/src/lib/libc/stdio/fgets.c:56
#1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at
/usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38
#2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow",
request=0xbfbfeb14)
at
/usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162
#3 0x28111540 in hosts_access (request=0xbfbfeb14) at
/usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132
#4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at
/usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843
(gdb) q
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/sbin/sshd, process 691
A few questions
1. Is this a known issue of any sort? I've done some searching on it,
but
haven't found anything of interest.
2. Should this be reported to FreeBSD bug tracker, or to libwrap (or
both)?
Basically, is FreeBSD's libwrap (more or less) in sync with the main
one, or
is it completely separate?
When given an input line of more than 2k bytes, libwrap ends up in an
infinite loop in xgets(), calling fgets() with a read length of one.
As fgets() reads the length minus one characters, it will keep
"reading" and returning zero length strings.
Thus your server processes will remain stuck until aborted.
This Q&D patch makes libwrap behave as documented in hosts_access(5):
--- misc.c.orig 2009-03-15 14:06:11.000000000 -0700
+++ misc.c 2009-03-15 14:06:49.000000000 -0700
@@ -48,6 +48,8 @@
ptr += got;
len -= got;
ptr[0] = 0;
+ if (len <= 1)
+ return start;
}
return (ptr > start ? start : 0);
}
The documented behavior is:
"An error is reported when ... when the length of an access control
rule exceeds the capacity of an internal buffer; ..."
This is only sligtly better, as the code will now try to parse the
remainder of the line as a rule, and either fail or, due to some
syntactic quirk, get a false match. From a security standpoint, both
are bad.
I don't think you'll get a false "allow" match in your case, but unless
you have a default "deny" rule somewhere at the end, access may be
granted when it shouldn't.
Please do file a FreeBSD bug. Is there even an upstream maintainer of
tcp wrappers? A quick search seems to indicate that it is more or
less abandoned, albeit adopted by several projects.
The immediate workarounds I can think of for you are:
- Somehow teach sshguard to write rules on multiple lines, each
shorter than 2k. Splitting lines using backslashes will not help,
as xgets() is concatenating continued lines into a single buffer
(the one that is too small) anyway.
- Apply the patch above, change the definition of BUFLEN in tcpdchk.c
and hosts_access.c to a "sufficiently large" value and rebuild
libwrap. Of course, there is no "sufficiently large" value; with
the current libwrap code, you'll always run the risk of lines being
too long.
The real fix involves rewriting chunks of the libwrap code, or finding
a version where someone has already done so.
$.02,
/Mikko
