liangkun9302 commented on issue #70496:
URL: https://github.com/apache/doris/issues/70496#issuecomment-6936480178

   Thank you @yiguolei, @Doris-Breakwater, @leeks8888, and @stinger1206 for 
your detailed analysis and suggestions.
   
   I really appreciate your help with this issue.
   
   Our production cluster is running Doris 4.1.5 (BE commit `917b20be838`), 
with 3 FE and 3 BE nodes. We previously experienced the same instability on 
4.1.4, and upgrading to 4.1.5 did not resolve it.
   
   The recurring BE crashes are significantly affecting production 
availability, so we would like to identify both a safe temporary workaround and 
the underlying root cause.
   
   ### 1. Regarding the suggested configuration changes
   
   Thank you @yiguolei for suggesting:
   
   - Setting `muzzy_decay_ms:0` in `JEMALLOC_CONF` in `be.conf`.
   - Setting `enable_memtable_on_sink_node=false`.
   
   We will evaluate these changes and, where applicable, test them in a 
controlled manner, since this is a production environment.
   
   Before proceeding, could you please clarify:
   
   - Is `muzzy_decay_ms:0` intended to mitigate a known jemalloc-related issue, 
or is it primarily a diagnostic workaround for possible memory corruption?
   - @leeks8888 mentioned that disabling `enable_memtable_on_sink_node` did not 
significantly reduce crashes in their environment. Would you still recommend 
this as a priority workaround for our case?
   - Are there any known side effects or performance impacts we should consider 
when applying these settings in production?
   
   ### 2. Possible shared memory corruption
   
   The ResourcePool `TaskMeta` corruption reported by @leeks8888 is 
particularly interesting.
   
   Our crashes occur in several different components, including:
   
   - `brpc::Stream::OnReceived()` — SIGABRT
   - `bthread::TaskGroup::sched_to()` — SIGSEGV
   - `TabletManager::find_best_tablets_to_compaction()` — SIGSEGV
   
   Although we cannot yet confirm that these crashes share the same root cause, 
the similar crash patterns across Doris 3.1.4 and 4.1.5 suggest that 
investigating possible memory corruption or object lifetime issues may be 
worthwhile.
   
   Could the maintainers help determine whether these failures might be related 
to a common underlying issue, rather than independent bugs?
   
   Also, could you confirm whether the official Doris 4.1.5 binary includes the 
relevant BRPC fixes, particularly `brpc#2949` and `brpc#3291`?
   
   ### 3. Additional diagnostics
   
   Most of our existing core dumps were marked as `truncated`.
   
   We will investigate the core dump configuration to improve the collection of 
complete crash data.
   
   Could you recommend the most useful additional diagnostics for identifying 
the root cause, especially for the BRPC and bthread crashes?
   
   We can provide sanitized logs, configuration details, and additional 
diagnostic information if needed.
   
   ### 4. Production workaround or fix
   
   Since all three BE nodes have experienced repeated crashes, this is a 
serious production stability issue for us.
   
   We would particularly appreciate guidance on whether:
   
   - There is an existing patch or newer build that addresses these crash 
signatures.
   - There is a recommended and tested workaround for production environments.
   - The issue should be investigated further by the Doris BE/BRPC maintainers.
   
   We are happy to cooperate with further troubleshooting and provide 
additional evidence.
   
   Thank you again for your time and support.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to