Regarding load growth, presumably you are referring to the load as reported
by JMX/nodetool. Have you actually looked at the disk utilization on the
nodes themselves? Potential issue I have seen:
http://www.mail-archive.com/user@cassandra.apache.org/msg18142.html

 

Dan

 

From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] 
Sent: November-03-11 14:40
To: user@cassandra.apache.org
Subject: Problem after upgrade to 1.0.1

 

I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go just
fine with the rolling upgrade.  But now I'm having extreme load growth on
one of my nodes (and others are growing faster than usual also).  I
attempted to run a cfstats against the extremely large node that was seeing
2x the load of others and I get this error below.  I'm also went into the
o.a.c.db.HintedHandoffManager mbean and attempted to list pending hints to
see if it was growing out of control for some reason, but that just times
out eventually for any node.  I'm not sure what to do next with this issue.

 

               Column Family: HintsColumnFamily

                SSTable count: 3

                Space used (live): 12681676437

                Space used (total): 10233130272

                Number of Keys (estimate): 384

                Memtable Columns Count: 117704

                Memtable Data Size: 115107307

                Memtable Switch Count: 66

                Read Count: 0

                Read Latency: NaN ms.

                Write Count: 21203290

                Write Latency: 0.014 ms.

                Pending Tasks: 0

                Key cache capacity: 3

                Key cache size: 0

                Key cache hit rate: NaN

                Row cache: disabled

                Compacted row minimum size: 30130993

                Compacted row maximum size: 9223372036854775807

Exception in thread "main" java.lang.IllegalStateException: Unable to
compute ceiling for max when histogram overflowed

        at
org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:1
70)

        at
org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395)

        at
org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStore.j
ava:293)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntr
ospector.java:93)

        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntr
ospector.java:27)

        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208
)

        at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)

        at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)

        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMB
eanServerInterceptor.java:666)

        at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)

        at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.
java:1404)

        at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.j
ava:72)

        at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMICon
nectionImpl.java:1265)

        at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConne
ctionImpl.java:1360)

        at
javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl
.java:600)

        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)

        at sun.rmi.transport.Transport$1.run(Transport.java:159)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)

        at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)

        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:
790)

        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:6
49)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:662)

 

Bryce Godfrey | Sr. Software Engineer |  <http://www.azaleos.com/> Azaleos
Corporation | T: 206.926.1978 | M: 206.849.2477

 

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.920 / Virus Database: 271.1.1/3993 - Release Date: 11/03/11
03:39:00

Reply via email to