I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go just fine
with the rolling upgrade. But now I'm having extreme load growth on one of my
nodes (and others are growing faster than usual also). I attempted to run a
cfstats against the extremely large node that was seeing 2x the load of others
and I get this error below. I'm also went into the
o.a.c.db.HintedHandoffManager mbean and attempted to list pending hints to see
if it was growing out of control for some reason, but that just times out
eventually for any node. I'm not sure what to do next with this issue.
Column Family: HintsColumnFamily
SSTable count: 3
Space used (live): 12681676437
Space used (total): 10233130272
Number of Keys (estimate): 384
Memtable Columns Count: 117704
Memtable Data Size: 115107307
Memtable Switch Count: 66
Read Count: 0
Read Latency: NaN ms.
Write Count: 21203290
Write Latency: 0.014 ms.
Pending Tasks: 0
Key cache capacity: 3
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 30130993
Compacted row maximum size: 9223372036854775807
Exception in thread "main" java.lang.IllegalStateException: Unable to compute
ceiling for max when histogram overflowed
at
org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:170)
at
org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395)
at
org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStore.java:293)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporation<http://www.azaleos.com/> | T: 206.926.1978 | M: 206.849.2477
