On Dec 29, 2012, at 8:53 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote:
> Can you post gc settings? Also check logs and see what it says These are the relevant jam settings: -home /usr/lib/jvm/j2re1.6-oracle/bin/../ -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8049M -Xmx8049M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss196k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true I have 3 servers (32GB RAM), with a RF of 3. I searched all of them for log messages related to a period when I had timeouts (19h20-19h30), only one of them showed messages for that timeframe, and none seem related with that CF: INFO [MemoryMeter:1] 2012-12-27 19:20:17,858 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='RevisionLog') liveRatio is 4.318314007200407 (just-counted was 4.318314007200407). calculation took 350ms for 8623 columns INFO [MemoryMeter:1] 2012-12-27 19:23:37,148 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='LinkPathsExist') liveRatio is 25.87012987012987 (just-counted was 25.87012987012987). calculation took 0ms for 10 columns INFO [MemoryMeter:1] 2012-12-27 19:28:32,736 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='BlockMetadata.BlockMetadata_used_idx') liveRatio is 1.7176206177506523 (just-counted was 1.7176206177506523). calculation took 62ms for 12941 columns INFO [MemoryMeter:1] 2012-12-27 19:30:12,752 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='Namespace') liveRatio is 20.097473571044617 (just-counted was 20.097473571044617). calculation took 10ms for 288 columns INFO [MemoryMeter:1] 2012-12-27 19:30:28,421 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='NamespaceDir') liveRatio is 4.801010311533358 (just-counted was 4.801010311533358). calculation took 96ms for 3138 columns > Also post how many writes and reads along with avg row size All rows have 3-6 counters. As for writes and reads: Column Family: UserQuotas SSTable count: 3 Space used (live): 2609839 Space used (total): 2609839 Number of Keys (estimate): 22016 Memtable Columns Count: 142705 Memtable Data Size: 768117 Memtable Switch Count: 26 Read Count: 822203 Read Latency: 0.305 ms. Write Count: 1024277 Write Latency: 0.066 ms. Pending Tasks: 0 Bloom Filter False Postives: 3 Bloom Filter False Ratio: 0.00000 Bloom Filter Space Used: 42584 Compacted row minimum size: 125 Compacted row maximum size: 770 Compacted row mean size: 298 Is there anything wrong with my configuration? Best regards, André Cruz