[
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100330#comment-17100330
]
Joey Lynch commented on CASSANDRA-15214:
----------------------------------------
> Since one should be able to trigger the OOM by looping allocating large chunk
> of memory, e.g. array, in the java code. What is the benefit of doing it so
> using jvmquake? I can see that in the killer_thread callback function, it
> also does long array allocation once notified by the gc callback.
Ah sorry I was not clear. I think the JVMStabilityDetector (which we call into
via inspectThrowable all over the place) should allocate the long array if we
see an OutOfMemoryError with message "Direct buffer memory", in turn triggering
a Heap OOM (which will trigger the normal resource exhausted mechanism). Since
we're not out of _heap_ memory we can trust that JVMStatbilityDetector can run.
I guess my proposal is to include jvmquake by default for linux deployments (I
can add more architectures if we want more, easy to opt out), and if
JVMStabilityDetector sees a "Direct buffer memory" OOM it should force the JVM
into a heap OOM, triggering jvmquake's resource exhausted handler.
This setup would guarantee that C* dies (and produces a heap dump) if any of
the following conditions hold:
* The JVM is out of heap memory
* The JVM has accumulated 30s of GC debt with 1:5 runtime weight (meaning that
we had <85% throughput for at least 30s): aka "GC spirals of death"
* The JVM is out of metaspace memory
* The JVM is out of threads
* (best effort, likely true) The JVM is out of native memory (so basically C*
is using 2x the heap size) -> triggers a heap oom -> triggers the first case
Unlike the built in JVM options jvmquake really actually works in these edge
cases (not only is there a test suite to prove it that the built in Java
options don't work but if you run inside the heap you fundamentally can't
guarantee you will run, e.g. why the kill -9 approach never really works).
> OOMs caught and not rethrown
> ----------------------------
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client, Messaging/Internode
> Reporter: Benedict Elliott Smith
> Priority: Normal
> Fix For: 4.0, 4.0-rc
>
> Attachments: oom-experiments.zip
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions,
> so presently there is no way to ensure that an OOM reaches the JVM handler to
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a
> single thread spawned at startup that waits for any exceptions we must
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed
> future proof approach, it may be worth paying the cost of a single thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
