stinger1206 commented on issue #70496:
URL: https://github.com/apache/doris/issues/70496#issuecomment-7033219298

   @liangkun9302 @leeks8888 @yiguolei  — sharing some findings from our 
investigation that might be relevant to your crashes.
   We've been investigating similar unexplained BE crashes in our production 
Doris cluster for several weeks and have some findings that might be relevant 
to your case.
   
   **Our environment:**
   - Apache Doris 4.1.5, coupled mode, 6 BEs on Kubernetes.
   - AlmaLinux 9.8.
   - Kernel before the upgrade: `5.14.0-687.5.4.el9_8`.
   - Kernel after the upgrade: `5.14.0-687.64.1.el9_8`.
   - jemalloc with Doris's default `muzzy_decay_ms:5000` configuration 
(`MADV_FREE`).
   - THP disabled, automatic NUMA balancing enabled.
   
   We observed crashes in different BE components, including brpc/bthread and 
bvar. We initially suspected a concurrency or object-lifetime issue in brpc.
   
   We spent several weeks analyzing core dumps, inspecting the relevant source 
code, building instrumented binaries, and running dedicated concurrency stress 
tests with ASan. Despite billions of operations and extensive forced bthread 
migration testing, we could not reproduce the suspected corruption through the 
migration path.
   
   **What eventually made a difference was upgrading the host kernel.**
   
   Between November 3 and November 6, we recorded 46 BE crashes. We performed a 
rolling kernel upgrade across our Kubernetes workers on November 5–6.
   
   Interestingly, each worker stopped experiencing these crashes after its 
individual kernel upgrade and reboot, while workers still running the old 
kernel continued crashing.
   
   The Doris BE binary remained unchanged.
   
   Following the upgrade, we observed no further crashes of this kind for more 
than 14 days, even after increasing the production workload by approximately 
3–5 times.
   
   **Possible kernel memory corruption**
   
   While investigating the kernel changelogs, we identified several 
memory-management fixes that could potentially explain this behavior. One 
particularly interesting candidate is:
   
   `x86/mm: Add missing saved dirty bit to page protection change mask` 
(RHEL-220787)
   
   This fix appears in the EL9.8 kernel changelog starting with 
`5.14.0-687.39.1.el9_8`.
   
   There has also been a recent investigation in the DuckDB community involving 
Linux kernel memory corruption associated with jemalloc and `MADV_FREE`:
   
   https://github.com/duckdb/duckdb/pull/28267
   
   Related kernel vulnerability:
   
   https://ubuntu.com/security/CVE-2026-97945
   
   The latter specifically involves transparent huge pages, so it is unlikely 
to be the exact mechanism in our environment, where THP is disabled. However, 
it demonstrates that allocator/kernel interactions can cause unexpected 
user-space memory corruption and misleading crash signatures in database 
engines.
   
   We haven't isolated the exact kernel patch responsible for resolving our 
crashes, so we cannot claim a definitive root cause yet. Nevertheless, the 
per-node correlation with the rolling kernel upgrade is quite compelling.
   
   Since your environment also uses an Enterprise Linux 9 kernel, I would 
strongly recommend testing a newer kernel on one of your BE hosts while keeping 
the Doris binary and configuration unchanged.
   
   It would be interesting to see whether a kernel upgrade has a similar effect 
in your environment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to