Hi Jeff,
I used below command to findout total off heap.
bin/nodetool cfstats | grep 'Off heap memory used' | cut -d ":" -f2 | awk
'{sum+=$1} END{print sum}'
*1417134538*
As per your suggestion, it is around 1GB only. we have around 50 tables
and 2 column family but still it doesn't make sense to having so much
resident memory.
On Mon, Nov 14, 2016 at 7:15 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com>
wrote:
> nodetool cfstats will show it per table.
>
>
>
> The bloom filter / compression data is typically (unless you have very
> unusual settings in your schema) 1-3GB each per TB of data, so with 235’ish
> GB/server, it’s unlikely bloom filter or compression data.
>
>
>
> The memTable is AT LEAST 1MB per columnfamily/table, so if you know how
> many tables you have, that may be an initial lower bound guess.
>
>
>
>
>
>
>
> *From: *ankit tyagi <ankittyagi.mn...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Date: *Sunday, November 13, 2016 at 11:33 PM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *Re: Too High resident memory of cassandra 2.2.8
>
>
>
> Hi Jeff,
>
> Below is the output of nodetool staus command.
>
>
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> -- Address Load Tokens Owns Host ID
> Rack
>
> UN 192.168.68.156 *235.79 GB* 256 ?
> e7b1a44d-0cd2-4b60-b322-4f989933fc51 rack1
>
> UN 192.168.68.157 *234.65 GB* 256 ?
> 70406f0b-3620-401e-beaa-15deb4b799ce rack1
>
> UN 192.168.69.146 256 ? d32e1e4d-ec86-4c3f-9397-11f37ff7b4d3
> rack1
>
> UN 192.168.69.147 *242.77 GB * 256 ?
> 646d9416-a467-4526-9656-959aa98404d0 rack1
>
> UN 192.168.69.148 *249.84 GB * 256 ?
> 9b0ab632-75f4-4781-a987-a00b8246ae97 rack1
>
> UN 192.168.69.149 *240.62 GB* 256 ?
> 406c4d3e-0933-4cba-935f-bfba16e6d878 rack1
>
>
>
>
>
> is there any command to find out the size of offheap memtable.
>
>
>
> On Mon, Nov 14, 2016 at 12:30 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com>
> wrote:
>
> Cassandra keeps certain data structures offheap, including bloom filters
> (scales with data size), compression metadata (scales with data size), and
> potentially memtables (scales with # of keyspaces/tables).
>
>
>
> How much data on your node? Onheap or offheap memtables?
>
>
>
>
>
>
>
> *From: *ankit tyagi <ankittyagi.mn...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Date: *Sunday, November 13, 2016 at 10:55 PM
> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
> *Subject: *Too High resident memory of cassandra 2.2.8
>
>
>
> Hi,
>
>
>
> we are using cassandra 2.2.8 version in production. we are seeing resident
> memory of cassndra process is very high 40G while heap size is only 8GB.
>
>
>
> root 23339 1 80 Nov11 ? 2-09:38:08 /opt/java8/bin/java
> -ea -javaagent:bin/../lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42* -Xms8192M -Xmx8192M
> -Xmn2048M* -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+PerfDisableSharedMem
> -XX:CompileCommandFile=bin/../conf/hotspot_compiler
> -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=10000
> -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:bin/../logs/gc.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
> -Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199
> -XX:+DisableExplicitGC -Djava.library.path=bin/../lib/sigar-bin
> -javaagent:/myntra/currentCassandra/lib/agent-1.2.jar=
> statsd.myntra.com:8125
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__statsd.myntra.com-3A8125&d=DgMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=0fYMi1mswzCVwZuipJxuigF790u8ULx3J5Py6llKbBQ&s=DEBDJVll8AVzBXyLsuXClwtqM3shJVQeZp4tNxNAWCc&e=>
> -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=bin/../logs
> -Dcassandra.storagedir=bin/../data -cp bin/../conf:bin/../build/
> classes/main:bin/../build/classes/thrift:bin/../lib/
> agent-1.2.jar:bin/../lib/airline-0.6.jar:bin/../lib/
> antlr-runtime-3.5.2.jar:bin/../lib/apache-cassandra-2.2.8.
> jar:bin/../lib/apache-cassandra-clientutil-2.2.8.jar:bin/../lib/apache-
> cassandra-thrift-2.2.8.jar:bin/../lib/cassandra-driver-
> core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:bin/../
> lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:
> bin/../lib/commons-lang3-3.1.jar:bin/../lib/commons-math3-
> 3.2.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/
> concurrentlinkedhashmap-lru-1.4.jar:bin/../lib/crc32ex-0.1.
> 1.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/ecj-4.4.2.
> jar:bin/../lib/guava-16.0.jar:bin/../lib/high-scale-lib-1.0.
> 6.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/
> jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.3.0.jar:bin/
> ../lib/java-statsd-client-3.1.0.jar:bin/../lib/javax.inject.
> jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jcl-over-slf4j-
> 1.7.7.jar:bin/../lib/jna-4.0.0.jar:bin/../lib/joda-time-2.
> 4.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-
> 0.9.2.jar:bin/../lib/log4j-over-slf4j-1.7.7.jar:bin/../
> lib/logback-classic-1.1.3.jar:bin/../lib/logback-core-1.1.3.
> jar:bin/../lib/lz4-1.3.0.jar:bin/../lib/metrics-core-2.2.0.
> jar:bin/../lib/metrics-core-3.1.0.jar:bin/../lib/metrics-
> jvm-3.1.0.jar:bin/../lib/metrics-logback-3.1.0.jar:bin/
> ../lib/metrics-statsd-2.3.0.jar:bin/../lib/netty-all-4.0.
> 23.Final.jar:bin/../lib/ohc-core-0.3.4.jar:bin/../lib/ohc-
> core-j8-0.3.4.jar:bin/../lib/reporter-config3-3.0.0.jar:
> bin/../lib/reporter-config-base-3.0.0.jar:bin/../lib/
> sigar-1.6.4.jar:bin/../lib/slf4j-api-1.7.7.jar:bin/../
> lib/snakeyaml-1.11.jar:bin/../lib/snappy-java-1.1.1.7.jar:
> bin/../lib/ST4-4.0.8.jar:bin/../lib/stream-2.5.2.jar:bin/../
> lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0.3.7.jar:bin/../lib/jsr223/*/*.jar
> org.apache.cassandra.service.CassandraDaemon
>
>
>
>
>
> below is the output of top command.
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 23339 root 20 0 295g * 39g 28g* S 107.5 63.5 3458:31 java
>
>
>
>
>
> *I couldn't understand why cassandra process is using so much resident
> memory while heap size is only 8gb. Any insight??*
>
>
>
