Public bug reported:
[ Impact ]
rasdaemon 0.8.4-1ubuntu0.1 aborts with SIGABRT ("*** buffer overflow
detected ***: terminated") when it processes a ras:non_standard_event.
Non-standard events are GHES/CPER records whose section type the kernel
does not decode itself, e.g. "general processor error" on x86 and
vendor-specific sections on arm64 (Ampere).
Consequences:
- The hardware error record that triggered the crash is not logged or
stored in the rasdaemon database.
- systemd restarts the service, but a burst of such events can reach the
unit's start limit and leave the machine without RAS monitoring.
Root cause: rasdaemon formats 16-byte GUIDs with three copies of the same
helper (uuid_le() in ras-non-standard-handler.c, uuid_le() in
ras-extlog-handler.c, uuid_be() in ras-cxl-handler.c). Each does:
p += snprintf(p, sizeof(uuid), "%.2x", (unsigned char)uu[...]);
The bound is always the full buffer size (37) instead of the space left
after p. Nothing really overflows (each call writes 2 characters), but
Ubuntu builds with _FORTIFY_SOURCE=3, so __snprintf_chk() knows the real
remaining size and aborts from the second iteration on (37 > 35 left).
In the non-standard handler uuid_le(ev.sec_type) runs before the event is
printed or stored, so every non_standard_event hits it regardless of
payload. The extlog and CXL helpers have the same defect, so
ras:extlog_mem_event (enabled on the affected host; its handler calls
uuid_le(ev->fru_id)) and the CXL events that call uuid_be() are expected
to abort the same way. I have only observed the non_standard_event crash.
Upstream bug: https://github.com/mchehab/rasdaemon/issues/260
Upstream fix:
https://github.com/mchehab/rasdaemon/commit/6202dfe17efe4d80ff1fc32773f3fa37cdb9da40
("core/types: add a safe version of UUID to str routine",
closes #260; replaces all three helpers)
Fixed upstream in v0.9.92 and v1.0.0. NOT in v0.8.4, v0.8.5 or v0.9.91.
[ Observed on ]
- Hardware: Intel Xeon Cascade Lake-SP (CPUID 0x50657, microcode
0x5003901), 48 CPUs, firmware-first (APEI GHES) error reporting
- Release: Ubuntu 26.04.1 LTS
- Kernel: linux 7.0.0-34.34 (amd64)
- rasdaemon 0.8.4-1ubuntu0.1, libtraceevent1 1:1.8.7-1
- The upstream reporter hit the same crash on arm64 (Ampere Altra) with
the Ubuntu 0.8.4-1 package, so it is not architecture-specific.
Sequence (2026-10-09, UTC):
12:58:33 mce_record: bank 0xc, status 0x8800004000300e0f, "COR LL Rx
detected CRC error - successful LLR without Phy Reinit" (UPI),
cpu 12, socketid 1, apicid 20. Decoded and stored normally.
12:58:48 Kernel logs a GHES record:
{1}[Hardware Error]: event severity: corrected
{1}[Hardware Error]: section_type: general processor error
{1}[Hardware Error]: error_type: 0x04 / bus error
{1}[Hardware Error]: processor_id: 0x0000000000000020
rasdaemon then logs "*** buffer overflow detected ***:
terminated" and dumps core (systemd restarts it).
Stack trace (systemd-coredump, no debug symbols):
#0 pthread_kill (libc.so.6 + 0xa61ac)
#1 raise (libc.so.6 + 0x45b7e)
#2 abort (libc.so.6 + 0x288ec)
#3 n/a (libc.so.6 + 0x29979)
#4 __fortify_fail (libc.so.6 + 0x143ed0)
#5 __chk_fail (libc.so.6 + 0x143844)
#6 __snprintf_chk (libc.so.6 + 0x145185)
#7 n/a (/usr/sbin/rasdaemon + 0x3c854)
#8 n/a (/usr/sbin/rasdaemon + 0x3cc56)
#9 n/a (libtraceevent.so.1 + 0x16304)
#10 tep_print_event (libtraceevent.so.1 + 0x173a6)
#11 n/a (/usr/sbin/rasdaemon + 0x1bc73)
#12 n/a (/usr/sbin/rasdaemon + 0x1c255)
#13 n/a (/usr/sbin/rasdaemon + 0x18ce5)
#14 n/a (libc.so.6 + 0x2a601)
#15 __libc_start_main (libc.so.6 + 0x2a718)
build-id: d5c8e73d44c0acc6fe1dc9196f22703d7a737630
Frames #7/#8 are inferred to be uuid_le() and
ras_non_standard_event_handler() from the 0.8.4 source; they were not
symbolized.
[ Test Plan ]
1. Root-cause reproducer (any amd64/arm64 Ubuntu, no special hardware).
NOTE: not yet run by the reporter; expected result stated below.
cat > repro.c <<'EOF'
#include <stdio.h>
static char *uuid_le(const char *uu)
{
static char uuid[sizeof("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx")];
char *p = uuid;
static const unsigned char le[16] =
{3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15};
for (int i = 0; i < 16; i++) {
p += snprintf(p, sizeof(uuid), "%.2x", (unsigned char)uu[le[i]]);
switch (i) { case 3: case 5: case 7: case 9: *p++ = '-'; }
}
*p = 0;
return uuid;
}
int main(void) { char g[16] = {0}; puts(uuid_le(g)); return 0; }
EOF
gcc -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 repro.c -o repro && ./repro
Expected with the bug: "*** buffer overflow detected ***", SIGABRT.
With sizeof(uuid) - (p - uuid) it prints
00000000-0000-0000-0000-000000000000.
2. Package test on hardware that emits non-standard GHES records:
a. Install the current package and confirm rasdaemon is running.
b. Trigger a non-standard CPER record: wait for a real event or, if the
platform supports APEI EINJ, inject a processor correctable error
(whether firmware reports it as a "general processor error" section
depends on the platform; check dmesg for "section_type").
c. Unfixed package: journalctl -u rasdaemon shows the abort and a core
dump.
d. Fixed package: rasdaemon keeps running, the journal shows a decoded
non_standard_event with a well-formed section type / fru id UUID,
and `ras-mc-ctl --errors` lists it.
3. Regression check: mc_event, mce_record and aer_event handling still
work and existing database entries still show in `ras-mc-ctl --errors`.
[ Where problems could occur ]
The patch changes only the size argument of snprintf() in three UUID
formatting helpers (non-standard, extlog, CXL). Output bytes are identical.
If the remaining-space calculation were wrong, the worst likely effect is a
truncated UUID in the non_standard_event, extlog_mem_event or CXL log
lines and database rows. With the fix, the non_standard_event and extlog
paths run to completion for the first time on affected systems, so any
latent problem there (for example database insert errors) would show up in
the journal for those event types only.
[ Other Info ]
- Upstream 6202dfe moves the helper into core/types, a layout 0.8.4 does
not have, so it will probably not cherry-pick. The attached patch is a
minimal backport covering all three sites.
- Workaround until fixed: disable the event after rasdaemon starts, e.g.
a systemd drop-in (untested, check the path exists):
[Service]
ExecStartPost=/bin/sh -c 'sleep 3; echo 0 >
/sys/kernel/tracing/instances/rasdaemon/events/ras/non_standard_event/enable'
Repeat for ras/extlog_mem_event if it is enabled. On x86 the same
errors usually also arrive through mce_record.
** Affects: rasdaemon (Ubuntu)
Importance: Undecided
Status: New
** Patch added: "patch was built against the upstream v0.8.4 tag"
https://bugs.launchpad.net/bugs/2172617/+attachment/6013805/+files/uuid-fortify-0.8.4.patch
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172617
Title:
rasdaemon aborts with "*** buffer overflow detected ***" on
ras:non_standard_event (uuid_le()/uuid_be() snprintf bound,
_FORTIFY_SOURCE=3)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rasdaemon/+bug/2172617/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
