Hi Ian, all,
Ian Chilton wrote on 18/08/2022 16:57:
We then run a "bird re-validate" cron job every hour (at twenty past the
hour):
/usr/sbin/birdc -s /run/bird-ipv6.ctl reload in all > /dev/null ;
/usr/sbin/birdc -s /run/bird-ipv4.ctl reload in all
Interestingly all 3 crashes have happened at just after twenty past the
hour, i.e soon after this cron job has run.
As you're running Bird 2.0.8 this should be no longer necessary. Per
2.0.8's release logs:
> Version 2.0.8 (2021-03-18)
> o Automatic channel reloads based on RPKI changes
So given all three crashes appear linked to this, stopping those manual
reloads should, hopefully, return you to stability.
You're also two bugfix releases behind. At INEX we've been running 2.0.9
for ~5/6 months now without issue.
There appears to be a lot of bugfixes between 2.0.8 and 2.0.10 so it
might be worthwhile updating or checking the git commit logs to see if
there's anything relevant to RPKI in there?
hth,
- Barry
It looks like the following in the logs:
Aug 17 17:20:01 rs1 CRON[29229]: (root) CMD (/usr/sbin/birdc -s
/run/bird-ipv6.ctl reload in all > /dev/null ; /usr/sbin/birdc -s
/run/bird-ipv4.ctl reload in all > /dev/null)
Aug 17 17:20:01 rs1 bird: Reloading protocol device1
Aug 17 17:20:01 rs1 bird: Reloading protocol pp_0121_asxx
..etc..
Aug 17 17:20:01 rs1 bird: Reloading protocol pp_1082_asxxxxxx
Aug 17 17:20:01 rs1 bird: Reloading protocol pb_1082_asxxxxxx
Aug 17 17:20:01 rs1 bird: Tagging invalid ROA 2001:xxxx:xxxx::/48 for
ASN xxxxx
..etc..
Aug 17 17:21:17 rs1 bird: Tagging invalid ROA x.x.x.x/23 for ASN xxxx
Aug 17 17:21:19 rs1 kernel: [7811815.959943] bird[586]: segfault at
f30021 ip 000055a1bf450fc3 sp 00007ffe64f3da98 error 4 in
bird[55a1bf42a000+d8000]
Aug 17 17:21:19 rs1 kernel: [7811815.966760] Code: 95 78 01 00 00 5b 5d
41 5c c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff b8 01 00 00 00
74 15 48 85 f6 0f 84 a6 00 00 00 <0f> b6 46 21 0f b6 57 21 29 d0 74 11
f3 c3 0f 1f 44 00 00 66 2e 0f
Aug 17 17:21:19 rs1 systemd[1]: bird-ipv4.service: Main process exited,
code=killed, status=11/SEGV
Aug 17 17:21:19 rs1 systemd[1]: bird-ipv4.service: Failed with result
'signal'.
Aug 17 17:21:19 rs1 systemd[1]: bird-ipv4.service: Service
RestartSec=100ms expired, scheduling restart.
Aug 17 17:21:19 rs1 systemd[1]: bird-ipv4.service: Scheduled restart
job, restart counter is at 1.
Aug 17 17:21:19 rs1 systemd[1]: Stopped BIRD - ipv4.
Aug 17 17:21:19 rs1 systemd[1]: Starting BIRD - ipv4...
Aug 17 17:21:22 rs1 systemd[1]: Started BIRD - ipv4.
Aug 17 17:21:22 rs1 bird: Started
When the second crash happened, we happened to be at RIPE84 so we
chatted to Maria in person. She said that it was possible to debug it,
but would need a core dump.
After looking in to this, I did:
ulimit -S -c unlimited
and installed the systemd-coredump package.
...which was supposed to dump a core file if a process crashed. I tested
this by killing a sleep command from the shell with kill -s 6 and it worked.
When the crash happened again yesterday, I hoped to have a core file to
send, but there is no sign of it having generated one :(
Testing on a test server, killing sleep generates a core file, but not
killing bird.
So two things - has anyone experienced similar crashes or have any ideas
why we might be seeing this?
Can anyone advise how to reliably get a core dump if bird crashes?
Thanks!
Ian
--
Kind regards,
Barry O'Donovan
Consultant
For and on behalf of INEX
https://www.inex.ie/support/
+353 1 531 3339
