Is compaction keeping up? Those replication settings are high - for every write, 10 nodes are doing something).
What other monitoring stats do you have - what is IO, CPU and network traffic like? Is the JVM GC activity growing? Anything else stick out like growing number of network connections to 9160 or 9042 on the cluster? The only leak-y things I've seen lately could either be: from a JVM bug leaking permgen from JMX invocation: https://issues.apache.org/jira/browse/CASSANDRA-6541 or if you have anything using thrift (this includes opscenter) and you are using HSHA as the rpc_server_type: https://issues.apache.org/jira/browse/CASSANDRA-4265 Otherwise, I think this is config/setup tuning. On Fri, May 23, 2014 at 7:00 AM, Alexey <alexe...@gmail.com> wrote: > Hi all, > > I've noticed increased latency on our tomcat REST-service (average 30ms, > max > 2sec). We are using Cassandra 1.2.16 with official DataStax Java > driver v1.0.3. > > Our setup: > > * 2 DCs > * each DC: 7 nodes > * RF=5 > * Leveled compaction > > After cassandra restart on all nodes, the latencies are alright again > (average < 5ms, max 50ms). > > Any thoughts are greatly appreciated. > > Thanks, > Alexey > -- ----------------- Nate McCall Austin, TX @zznate Co-Founder & Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
