Package: iptables Version: 1.8.4-3 In our cockpit CI tests on debian-testing I noticed an awful lot of tests that fail due to an iptables crash. (The tests themselves are fine, but we fail tests on unexpected journal messages, such as this one).
coredumpctl shows the following meta-info:
PID: 2020 (iptables)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Mon 2020-02-17 08:34:38 UTC (6min ago)
Command Line: /usr/sbin/iptables -w -L -n
Executable: /usr/sbin/xtables-nft-multi
Control Group: /system.slice/libvirtd.service
Unit: libvirtd.service
Slice: system.slice
Boot ID: faf0d59c49e6446580b7f2ce9d19c1a3
Machine ID: 7cb7efc599dc4bf0a81ebee56065e42f
Hostname: debian
Storage:
/var/lib/systemd/coredump/core.iptables.0.faf0d59c49e6446580b7f2ce9d19c1a3.2020.1581928478000000000000.lz4
Message: Process 2020 (iptables) of user 0 dumped core.
Stack trace of thread 2020:
#0 0x00007f216f7a07c0 nftnl_rule_lookup_byindex
(libnftnl.so.11 + 0xe7c0)
#1 0x00005617956f6e44 n/a (xtables-nft-multi + 0x19e44)
#2 0x00005617956f714b n/a (xtables-nft-multi + 0x1a14b)
#3 0x00005617956ed149 n/a (xtables-nft-multi + 0x10149)
#4 0x00007f216f7a0aa5 nftnl_chain_list_foreach (libnftnl.so.11
+ 0xeaa5)
#5 0x00005617956efa13 n/a (xtables-nft-multi + 0x12a13)
#6 0x00005617956efa32 n/a (xtables-nft-multi + 0x12a32)
#7 0x00005617956efab7 n/a (xtables-nft-multi + 0x12ab7)
#8 0x00005617956ead65 n/a (xtables-nft-multi + 0xdd65)
#9 0x00005617956e9072 n/a (xtables-nft-multi + 0xc072)
#10 0x00005617956e91ba n/a (xtables-nft-multi + 0xc1ba)
#11 0x00007f216f5e5bbb __libc_start_main (libc.so.6 + 0x26bbb)
#12 0x00005617956e612a n/a (xtables-nft-multi + 0x912a)
I installed the dbgsym packages and generated a full backtrace [2]. However, at
first sight this doesn't look too useful, other than perhaps that
nftnl_rule_lookup_byindex() is called with a NULL value of "c". I attach the
compressed core dump, in case that's useful.
Installed packages:
ii docker.io 19.03.5+dfsg1-2 amd64 Linux
container runtime
ii firewalld 0.8.1-1 all dynamically
managed firewall with support for network zones
ii iptables 1.8.4-3 amd64 administration
tools for packet filtering and NAT
ii libnftables1:amd64 0.9.3-2 amd64 Netfilter
nftables high level userspace API library
ii libnftnl11:amd64 1.1.5-1 amd64 Netfilter
nftables userspace API library
ii linux-image-5.4.0-3-cloud-amd64 5.4.13-1 amd64 Linux 5.4 for
x86-64 cloud (signed)
Notably, nftables is *not* installed. I'm not actually sure why, as iptables
recommends it and our image setup script [3] does not use
--no-install-recommends. I also don't know if nftables will even affect this.
We don't do any iptables configuration on our image (again, see [3]), the only
thing that it does is install "firewalld", which then pulls in iptables.
I haven't yet found a simple CLI way how to reproduce this, I'll keep looking.
For now, I attach the complete journal of that boot, it may reveal some
interesting interactions between firewalld, docker, and iptables.
Thanks,
Martin
[1]
https://logs.cockpit-project.org/logs/pull-532-20200217-072121-d7ba954f-debian-testing-cockpit-project-cockpit/log.html
[2]
#0 nftnl_rule_lookup_byindex (c=c@entry=0x0, index=index@entry=0) at
chain.c:863
__mptr = <optimized out>
r = <optimized out>
#1 0x00005617956f6e44 in nft_rule_list_update (c=0x0, data=0x7ffee5d02ad0) at
nft-cache.c:391
h = 0x7ffee5d02ad0
buf = <optimized out>
nlh = <optimized out>
rule = <optimized out>
ret = <optimized out>
__PRETTY_FUNCTION__ = "nft_rule_list_update"
#2 0x00005617956f714b in fetch_rule_cache (chain=0x56179740d830
"`\327@\227\027V", t=0x56179570fc80 <xtables_ipv4>, h=0x7ffee5d02ad0) at
nft-cache.c:432
list = <optimized out>
c = <optimized out>
i = <optimized out>
i = <optimized out>
list = <optimized out>
c = <optimized out>
type = <optimized out>
#3 __nft_build_cache (h=h@entry=0x7ffee5d02ad0,
level=level@entry=NFT_CL_RULES, t=0x56179570fc80 <xtables_ipv4>, set=0x0,
chain=0x56179740d830 "`\327@\227\027V")
at nft-cache.c:482
genid_start = 69
genid_stop = 69
#4 0x00005617956f730d in __nft_build_cache (chain=<optimized out>,
set=<optimized out>, t=<optimized out>, level=<optimized out>, h=<optimized
out>)
at nft-cache.c:515
genid_start = <optimized out>
genid_stop = <optimized out>
genid_start = <optimized out>
genid_stop = <optimized out>
#5 nft_build_cache (h=h@entry=0x561795702c68, c=c@entry=0x7ffee5d02ad0) at
nft-cache.c:515
t = <optimized out>
table = <optimized out>
chain = <optimized out>
#6 0x00005617956ed149 in nft_is_chain_compatible (c=0x7ffee5d02ad0,
data=0x561795702c68) at nft.c:3303
table = <optimized out>
chain = <optimized out>
tname = <optimized out>
cname = <optimized out>
type = <optimized out>
h = 0x561795702c68
hook = <optimized out>
prio = <optimized out>
#7 0x00007f216f7a0aa5 in nftnl_chain_list_foreach (chain_list=0x56179740b550,
cb=cb@entry=0x5617956ed130 <nft_is_chain_compatible>,
data=data@entry=0x7ffee5d02ad0) at chain.c:1011
cur = <optimized out>
tmp = 0x56179740d870
ret = <optimized out>
#8 0x00005617956efa13 in nft_is_table_compatible (h=0x7ffee5d02ad0,
table=table@entry=0x561795702c68 "filter", chain=chain@entry=0x0) at nft.c:3341
clist = <optimized out>
#9 0x00005617956efa32 in nft_assert_table_compatible (h=<optimized out>,
table=0x561795702c68 "filter", chain=0x0) at nft.c:3352
pfx = 0x561795702338 ""
sfx = 0x561795702338 ""
#10 0x00005617956efab7 in nft_rule_list (h=h@entry=0x7ffee5d02ad0, chain=0x0,
table=0x561795702c68 "filter", rulenum=0, format=15) at nft.c:2358
ops = 0x561795711560 <nft_family_ops_ipv4>
list = <optimized out>
iter = <optimized out>
c = <optimized out>
found = false
#11 0x00005617956ead65 in list_entries (linenumbers=<optimized out>,
expanded=<optimized out>, numeric=<optimized out>, verbose=<optimized out>,
rulenum=<optimized out>, table=<optimized out>, chain=<optimized out>,
h=0x7ffee5d02ad0) at xtables.c:527
format = <optimized out>
format = <optimized out>
#12 do_commandx (h=h@entry=0x7ffee5d02ad0, argc=argc@entry=4,
argv=argv@entry=0x7ffee5d02d68, table=table@entry=0x7ffee5d02ac8,
restore=restore@entry=false)
at xtables.c:1102
ret = 1
p = {command = 32, rulenum = 0, table = 0x561795702c68 "filter", chain
= 0x0, newname = 0x0, policy = 0x0, restore = false, verbose = 0, xlate = false}
cs = {{eb = {bitmask = 0, invflags = 0, ethproto = 0, in = '\000'
<repeats 15 times>, logical_in = '\000' <repeats 15 times>,
out = '\000' <repeats 15 times>, logical_out = '\000' <repeats 15
times>, sourcemac = "\000\000\000\000\000", sourcemsk = "\000\000\000\000\000",
destmac = "\000\000\000\000\000", destmsk =
"\000\000\000\000\000"}, fw = {ip = {src = {s_addr = 0}, dst = {s_addr = 0},
smsk = {s_addr = 0},
dmsk = {s_addr = 0}, iniface = '\000' <repeats 15 times>,
outiface = '\000' <repeats 15 times>, iniface_mask = '\000' <repeats 15 times>,
outiface_mask = '\000' <repeats 15 times>, proto = 0, flags = 0
'\000', invflags = 0 '\000'}, nfcache = 0, target_offset = 0, next_offset = 0,
comefrom = 0, counters = {pcnt = 0, bcnt = 0}, elems =
0x7ffee5d029c0 ""}, fw6 = {ipv6 = {src = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
0, 0, 0}}}, dst = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
0, 0, 0}}}, smsk = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
0, 0, 0}}}, dmsk = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
0, 0, 0}}}, iniface = '\000' <repeats 15 times>,
outiface = '\000' <repeats 15 times>, iniface_mask = '\000'
<repeats 15 times>, outiface_mask = '\000' <repeats 15 times>, proto = 0,
tos = 0 '\000', flags = 0 '\000', invflags = 0 '\000'}, nfcache
= 0, target_offset = 0, next_offset = 0, comefrom = 0, counters = {pcnt = 0,
bcnt = 0}, elems = 0x7ffee5d029f8 ""}, arp = {arp = {src =
{s_addr = 0}, tgt = {s_addr = 0}, smsk = {s_addr = 0}, tmsk = {s_addr = 0},
arhln = 0 '\000', arhln_mask = 0 '\000', src_devaddr = {addr =
'\000' <repeats 15 times>, mask = '\000' <repeats 15 times>}, tgt_devaddr = {
addr = '\000' <repeats 15 times>, mask = '\000' <repeats 15
times>}, arpop = 0, arpop_mask = 0, arhrd = 0, arhrd_mask = 0, arpro = 0,
arpro_mask = 0, iniface = '\000' <repeats 15 times>, outiface =
'\000' <repeats 15 times>, iniface_mask = '\000' <repeats 15 times>,
outiface_mask = '\000' <repeats 15 times>, flags = 0 '\000',
invflags = 0}, target_offset = 0, next_offset = 0, comefrom = 0, counters = {
pcnt = 0, bcnt = 0}, elems = 0x7ffee5d02a10 ""}}, invert = 0, c
= -1, options = 1, matches = 0x0, match_list = 0x0, target = 0x0, counters = {
pcnt = 0, bcnt = 0}, protocol = 0x0, proto_used = 0, jumpto =
0x561795702338 "", argv = 0x7ffee5d02d68, restore = false}
args = {family = 2, proto = 0, flags = 0 '\000', invflags = 0 '\000',
iniface = '\000' <repeats 15 times>, outiface = '\000' <repeats 15 times>,
iniface_mask = '\000' <repeats 15 times>, outiface_mask = '\000'
<repeats 15 times>, goto_set = false, shostnetworkmask = 0x0, dhostnetworkmask
= 0x0,
pcnt = 0x0, bcnt = 0x0, s = {addr = {v4 = 0x0, v6 = 0x0}, naddrs = 0,
mask = {v4 = 0x0, v6 = 0x0}}, d = {addr = {v4 = 0x0, v6 = 0x0}, naddrs = 0,
mask = {v4 = 0x0, v6 = 0x0}}, pcnt_cnt = 0, bcnt_cnt = 0}
#13 0x00005617956e9072 in xtables_main (family=family@entry=2,
progname=progname@entry=0x561795702011 "iptables", argc=4, argv=0x7ffee5d02d68)
at xtables-standalone.c:72
ret = <optimized out>
table = 0x561795702c68 "filter"
h = {family = 2, nl = 0x56179740b2a0, nlsndbuffsiz = 0, nlrcvbuffsiz =
0, portid = 2020, seq = 0, nft_genid = 68, rule_id = 0, obj_list = {
next = 0x56179740b320, prev = 0x56179740b320}, obj_list_num = 1,
batch = 0x0, err_list = {next = 0x7ffee5d02b18, prev = 0x7ffee5d02b18},
ops = 0x561795711560 <nft_family_ops_ipv4>, tables = 0x56179570fc80
<xtables_ipv4>, cache_index = 0, __cache = {{tables = 0x56179740b370, table = {{
chains = 0x56179740e530, sets = 0x56179740b530, initialized =
true}, {chains = 0x0, sets = 0x0, initialized = false}, {chains = 0x0,
sets = 0x0, initialized = false}, {chains = 0x0, sets = 0x0,
initialized = false}, {chains = 0x0, sets = 0x0, initialized = false}}}, {
tables = 0x0, table = {{chains = 0x0, sets = 0x0, initialized =
false}, {chains = 0x0, sets = 0x0, initialized = false}, {chains = 0x0, sets =
0x0,
initialized = false}, {chains = 0x0, sets = 0x0, initialized
= false}, {chains = 0x0, sets = 0x0, initialized = false}}}},
cache = 0x7ffee5d02b40, cache_level = NFT_CL_NONE, restore = false,
noflush = false, config_done = 0 '\000', error = {lineno = 0}}
#14 0x00005617956e91ba in xtables_ip4_main (argc=<optimized out>,
argv=<optimized out>) at xtables-standalone.c:96
No locals.
#15 0x00007f216f5e5bbb in __libc_start_main (main=0x5617956e60f0 <main>,
argc=4, argv=0x7ffee5d02d68, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7ffee5d02d58) at
../csu/libc-start.c:308
self = <optimized out>
result = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -2667567350696714500,
94659291275520, 140732754046304, 0, 0, -8563589328024183044,
-8604388788316348676},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7ffee5d02d90,
0x7f216f806190}, data = {prev = 0x0, cleanup = 0x0, canceltype = -439341680}}}
not_first_call = <optimized out>
#16 0x00005617956e612a in _start ()
No symbol table info available.
[3]
https://github.com/cockpit-project/bots/blob/master/images/scripts/debian.setup
xtables-nft-multi.core.xz
Description: application/xz
journal.txt.xz
Description: application/xz
