The first thing I look for with timeouts like that is a flush storm causing blocking in the write path (due to the internal "switch lock").
Take a look in the logs, for a number of messages such as "enqueuing CF…" and "writing cf..". Look for a pattern of enqueuing cf messages that occur immediately after writing cf messages. Hope that helps. ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/01/2013, at 4:25 AM, André Cruz <andre.c...@co.sapo.pt> wrote: > Does anyone see anything wrong in these settings? Anything to account for a > 8s timeout during a counter increment? > > Thanks, > André > > On 31/12/2012, at 14:35, André Cruz <andre.c...@co.sapo.pt> wrote: > >> On Dec 29, 2012, at 8:53 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: >> >>> Can you post gc settings? Also check logs and see what it says >> >> These are the relevant jam settings: >> >> -home /usr/lib/jvm/j2re1.6-oracle/bin/../ >> -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar >> -XX:+UseThreadPriorities >> -XX:ThreadPriorityPolicy=42 >> -Xms8049M >> -Xmx8049M >> -Xmn800M >> -XX:+HeapDumpOnOutOfMemoryError >> -Xss196k >> -XX:+UseParNewGC >> -XX:+UseConcMarkSweepGC >> -XX:+CMSParallelRemarkEnabled >> -XX:SurvivorRatio=8 >> -XX:MaxTenuringThreshold=1 >> -XX:CMSInitiatingOccupancyFraction=75 >> -XX:+UseCMSInitiatingOccupancyOnly >> -Djava.net.preferIPv4Stack=true >> >> I have 3 servers (32GB RAM), with a RF of 3. I searched all of them for log >> messages related to a period when I had timeouts (19h20-19h30), only one of >> them showed messages for that timeframe, and none seem related with that CF: >> >> INFO [MemoryMeter:1] 2012-12-27 19:20:17,858 Memtable.java (line 213) >> CFS(Keyspace='Disco', ColumnFamily='RevisionLog') liveRatio is >> 4.318314007200407 (just-counted was 4.318314007200407). calculation took >> 350ms for 8623 columns >> INFO [MemoryMeter:1] 2012-12-27 19:23:37,148 Memtable.java (line 213) >> CFS(Keyspace='Disco', ColumnFamily='LinkPathsExist') liveRatio is >> 25.87012987012987 (just-counted was 25.87012987012987). calculation took >> 0ms for 10 columns >> INFO [MemoryMeter:1] 2012-12-27 19:28:32,736 Memtable.java (line 213) >> CFS(Keyspace='Disco', ColumnFamily='BlockMetadata.BlockMetadata_used_idx') >> liveRatio is 1.7176206177506523 (just-counted was 1.7176206177506523). >> calculation took 62ms for 12941 columns >> INFO [MemoryMeter:1] 2012-12-27 19:30:12,752 Memtable.java (line 213) >> CFS(Keyspace='Disco', ColumnFamily='Namespace') liveRatio is >> 20.097473571044617 (just-counted was 20.097473571044617). calculation took >> 10ms for 288 columns >> INFO [MemoryMeter:1] 2012-12-27 19:30:28,421 Memtable.java (line 213) >> CFS(Keyspace='Disco', ColumnFamily='NamespaceDir') liveRatio is >> 4.801010311533358 (just-counted was 4.801010311533358). calculation took >> 96ms for 3138 columns >> >> >>> Also post how many writes and reads along with avg row size >> >> All rows have 3-6 counters. As for writes and reads: >> >> Column Family: UserQuotas >> SSTable count: 3 >> Space used (live): 2609839 >> Space used (total): 2609839 >> Number of Keys (estimate): 22016 >> Memtable Columns Count: 142705 >> Memtable Data Size: 768117 >> Memtable Switch Count: 26 >> Read Count: 822203 >> Read Latency: 0.305 ms. >> Write Count: 1024277 >> Write Latency: 0.066 ms. >> Pending Tasks: 0 >> Bloom Filter False Postives: 3 >> Bloom Filter False Ratio: 0.00000 >> Bloom Filter Space Used: 42584 >> Compacted row minimum size: 125 >> Compacted row maximum size: 770 >> Compacted row mean size: 298 >> >> >> Is there anything wrong with my configuration? >> >> Best regards, >> André Cruz >> >>
