在 2025/7/24 21:34, Breno Leitao 写道:
Hello Shuai,
On Thu, Jul 24, 2025 at 04:00:09PM +0800, Shuai Xue wrote:
在 2025/7/23 00:56, Breno Leitao 写道:
Introduce a generic infrastructure for tracking recoverable hardware
errors (HW errors that did not cause a panic) and record them for vmcore
consumption. This aids post-mortem crash analysis tools by preserving
a count and timestamp for the last occurrence of such errors.
Add centralized logging for three common sources of recoverable hardware
errors:
The term "recoverable" is highly ambiguous. Even within the x86
architecture, different vendors define errors differently. I'm not
trying to be pedantic about classification. As far as I know, for 2-bit
memory errors detected by scrub, AMD defines them as deferred errors
(DE) and handles them with log_error_deferred, while Intel uses
machine_check_poll. For 2-bit memory errors consumed by processes,
both Intel and AMD use MCE handling via do_machine_check(). Does your
HWERR_RECOV_MCE only focus on synchronous UE errors handled in
do_machine_check? What makes it special?
I understand that deferred errors (DE) detected by memory scrubbing are
typically silent and may not significantly impact system stability. In
other words, I’m not convinced that including DE metrics in crash dumps
would be helpful for correlating crashes with hardware issues—it might
just add noise.
Do you think it would be valuable to also log these events within
log_error_deferred()?
Not really, as you meationed, the DE is typically silent in backgroud.
But I hope it is well documented.
- if (ghes_severity(estatus->error_severity) >= GHES_SEV_PANIC)
+ sev = ghes_severity(estatus->error_severity);
+ if (sev == GHES_SEV_RECOVERABLE || sev == GHES_SEV_CORRECTED)
+ hwerr_log_error_type(HWERR_RECOV_GHES);
APEI does not define an error type named GHES. GHES is just a kernel
driver name. Many hardware error types can be handled in GHES (see
ghes_do_proc), for example, AER is routed by GHES when firmware-first
mode is used. As far as I know, firmware-first mode is commonly used in
production. Should GHES errors be categorized into AER, memory, and CXL
memory instead?
I also considered slicing the data differently initially, but then
realized it would add more complexity than necessary for my needs.
If you believe we should further subdivide the data, I’m happy to do so.
You’re suggesting a structure like this, which would then map to the
corresponding CPER_SEC_ sections:
enum hwerr_error_type {
HWERR_RECOV_AER, // maps to CPER_SEC_PCIE
HWERR_RECOV_MCE, // maps to default MCE + CPER_SEC_PCIE
CPER_SEC_PCIE is typo?
HWERR_RECOV_CXL, // maps to CPER_SEC_CXL_*
HWERR_RECOV_MEMORY, // maps to CPER_SEC_PLATFORM_MEM
}
Additionally, what about events related to CPU, Firmware, or DMA
errors—for example, CPER_SEC_PROC, CPER_SEC_FW, CPER_SEC_DMAR? Should we
include those in the classification as well?
I would like to split a error from ghes to its own type,
it sounds more reasonable. I can not tell what happened from HWERR_RECOV_AERat
all :(
Thanks for your review and for the ongoing discussion!
--breno
Thanks.
Shuai
