Dmesg will often print a message saying that it had to kill a process if the server was short of memory, so you will have to dump the output to a file and check. If a process is killed to reclaim memory for the system, then it will dump a list of all processes and the actual process that was killed. So maybe, you can check for a kill like this - "dmesg | grep -i kill" If you do find a line (or two), then you need to examine the output carefully.
In production, I tend to dump a lot of GC output also which helps troubleshooting. E.g. Below is what I have. If you look, I also have a flag that says that if the heap runs out of memory (which is rare), then dump files. If dmesg does not show your processes being killed, then you may have to dump gc logging info to get some insight. -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -XX:MaxTenuringThreshold=2 -XX:SurvivorRatio=8 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768 -XX:NewSize=750m -XX:MaxNewSize=750m -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -Xloggc:<some-path-for-gc-logs> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=<true|false> -Dcom.sun.management.jmxremote.authenticate=<true|false> From: Abhishek Kumar Maheshwari <abhishek.maheshw...@timesinternet.in> Date: Wednesday, March 22, 2017 at 5:18 PM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: RE: [Cassandra 3.0.9] Cannot allocate memory JVM config is as below: -Xms16G -Xmx16G -Xmn3000M What I need to check in dmesg? From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com] Sent: 23 March 2017 03:39 To: Abhishek Kumar Maheshwari <abhishek.maheshw...@timesinternet.in>; user@cassandra.apache.org Subject: RE: [Cassandra 3.0.9] Cannot allocate memory And what is the configured max heap? Sometimes you may also be able to see some useful messages in "dmesg" output. Jayesh ________________________________ From: Abhishek Kumar Maheshwari <abhishek.maheshw...@timesinternet.in<mailto:abhishek.maheshw...@timesinternet.in>> Sent: Wednesday, March 22, 2017 5:05:14 PM To: Thakrar, Jayesh; user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: RE: [Cassandra 3.0.9] Cannot allocate memory No only Cassandra is running on these servers. From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com] Sent: 22 March 2017 22:27 To: Abhishek Kumar Maheshwari <abhishek.maheshw...@timesinternet.in<mailto:abhishek.maheshw...@timesinternet.in>>; user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: [Cassandra 3.0.9] Cannot allocate memory Is/are the Cassandra server(s) shared? E.g. do they run mesos + spark? From: Abhishek Kumar Maheshwari <abhishek.maheshw...@timesinternet.in<mailto:abhishek.maheshw...@timesinternet.in>> Date: Wednesday, March 22, 2017 at 12:45 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: [Cassandra 3.0.9] Cannot allocate memory Hi all, I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches. Hardware info: Ram 64 GB. Core: 40 Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory. Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) [thread 140033204860672 also had an error] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) [thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) Please let me know what I miss? Thanks & Regards, Abhishek Kumar Maheshwari +91- 9999805591 (Mobile) Times Internet Ltd. | A Times of India Group Company FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA P Please do not print this email unless it is absolutely necessary. Spread environmental awareness. YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
