nodetool cfstats will show it per table.
The bloom filter / compression data is typically (unless you have very unusual
settings in your schema) 1-3GB each per TB of data, so with 235’ish GB/server,
it’s unlikely bloom filter or compression data.
The memTable is AT LEAST 1MB per columnfamily/table, so if you know how many
tables you have, that may be an initial lower bound guess.
From: ankit tyagi <ankittyagi.mn...@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Sunday, November 13, 2016 at 11:33 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Too High resident memory of cassandra 2.2.8
Hi Jeff,
Below is the output of nodetool staus command.
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.68.156 235.79 GB 256 ?
e7b1a44d-0cd2-4b60-b322-4f989933fc51 rack1
UN 192.168.68.157 234.65 GB 256 ?
70406f0b-3620-401e-beaa-15deb4b799ce rack1
UN 192.168.69.146 256 ? d32e1e4d-ec86-4c3f-9397-11f37ff7b4d3
rack1
UN 192.168.69.147 242.77 GB 256 ?
646d9416-a467-4526-9656-959aa98404d0 rack1
UN 192.168.69.148 249.84 GB 256 ?
9b0ab632-75f4-4781-a987-a00b8246ae97 rack1
UN 192.168.69.149 240.62 GB 256 ?
406c4d3e-0933-4cba-935f-bfba16e6d878 rack1
is there any command to find out the size of offheap memtable.
On Mon, Nov 14, 2016 at 12:30 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote:
Cassandra keeps certain data structures offheap, including bloom filters
(scales with data size), compression metadata (scales with data size), and
potentially memtables (scales with # of keyspaces/tables).
How much data on your node? Onheap or offheap memtables?
From: ankit tyagi <ankittyagi.mn...@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Sunday, November 13, 2016 at 10:55 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Too High resident memory of cassandra 2.2.8
Hi,
we are using cassandra 2.2.8 version in production. we are seeing resident
memory of cassndra process is very high 40G while heap size is only 8GB.
root 23339 1 80 Nov11 ? 2-09:38:08 /opt/java8/bin/java -ea
-javaagent:bin/../lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M
-Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB -XX:+PerfDisableSharedMem
-XX:CompileCommandFile=bin/../conf/hotspot_compiler -XX:CMSWaitDuration=10000
-XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways
-XX:CMSWaitDuration=10000 -XX:+UseCondCardMark -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure
-Xloggc:bin/../logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M -Djava.net.preferIPv4Stack=true
-Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC
-Djava.library.path=bin/../lib/sigar-bin
-javaagent:/myntra/currentCassandra/lib/agent-1.2.jar=statsd.myntra.com:8125
-Dlogback.configurationFile=logback.xml -Dcassandra.logdir=bin/../logs
-Dcassandra.storagedir=bin/../data -cp
bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/agent-1.2.jar:bin/../lib/airline-0.6.jar:bin/../lib/antlr-runtime-3.5.2.jar:bin/../lib/apache-cassandra-2.2.8.jar:bin/../lib/apache-cassandra-clientutil-2.2.8.jar:bin/../lib/apache-cassandra-thrift-2.2.8.jar:bin/../lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang3-3.1.jar:bin/../lib/commons-math3-3.2.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:bin/../lib/crc32ex-0.1.1.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/ecj-4.4.2.jar:bin/../lib/guava-16.0.jar:bin/../lib/high-scale-lib-1.0.6.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.3.0.jar:bin/../lib/java-statsd-client-3.1.0.jar:bin/../lib/javax.inject.jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jcl-over-slf4j-1.7.7.jar:bin/../lib/jna-4.0.0.jar:bin/../lib/joda-time-2.4.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.9.2.jar:bin/../lib/log4j-over-slf4j-1.7.7.jar:bin/../lib/logback-classic-1.1.3.jar:bin/../lib/logback-core-1.1.3.jar:bin/../lib/lz4-1.3.0.jar:bin/../lib/metrics-core-2.2.0.jar:bin/../lib/metrics-core-3.1.0.jar:bin/../lib/metrics-jvm-3.1.0.jar:bin/../lib/metrics-logback-3.1.0.jar:bin/../lib/metrics-statsd-2.3.0.jar:bin/../lib/netty-all-4.0.23.Final.jar:bin/../lib/ohc-core-0.3.4.jar:bin/../lib/ohc-core-j8-0.3.4.jar:bin/../lib/reporter-config3-3.0.0.jar:bin/../lib/reporter-config-base-3.0.0.jar:bin/../lib/sigar-1.6.4.jar:bin/../lib/slf4j-api-1.7.7.jar:bin/../lib/snakeyaml-1.11.jar:bin/../lib/snappy-java-1.1.1.7.jar:bin/../lib/ST4-4.0.8.jar:bin/../lib/stream-2.5.2.jar:bin/../lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0.3.7.jar:bin/../lib/jsr223/*/*.jar
org.apache.cassandra.service.CassandraDaemon
below is the output of top command.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23339 root 20 0 295g 39g 28g S 107.5 63.5 3458:31 java
I couldn't understand why cassandra process is using so much resident memory
while heap size is only 8gb. Any insight??
smime.p7s
Description: S/MIME cryptographic signature
