I don't have access to the test system anymore. We did move to lower number of CFs and dont see this problem any more. I remember when I noticed the size in system.log it was little more than UINT_MAX (4294967295). I was able to recreate it multiple times. So I am wondering if there are any stats counters in the system which is set to unsigned int instead of unsigned long?
thanks Ramesh On Tue, Oct 4, 2011 at 3:20 AM, aaron morton <aa...@thelastpickle.com> wrote: > That row has a size of 819 peta bytes, so something is odd there. The error > is a result of that value been so huge. When you rant he same script on > 0.8.6 what was the max size of the Migrations CF ? > As Jonathan says, it's unlikely anyone would have tested creating 5000 CF's. > Most people only create a few 10's of CF's at most. > either use fewer CF's or… > * dump the Migrations CF using sstable2json to take a look around > * work out steps to reproduce and report it on Jira > Hope that helps. > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > On 4/10/2011, at 11:30 AM, Ramesh Natarajan wrote: > > We recreated the schema using the same input file on both clusters and they > are running identical load. > Isn't the exception thrown in the system CF? > this line looks strange: > Compacted row maximum size: 9223372036854775807 > thanks > Ramesh > > On Mon, Oct 3, 2011 at 5:26 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >> >> Looks like you have unexpectedly large rows in your 1.0 cluster but >> not 0.8. I guess you could use sstable2json to manually check your >> row sizes. >> >> On Mon, Oct 3, 2011 at 5:20 PM, Ramesh Natarajan <rames...@gmail.com> >> wrote: >> > It happens all the time on 1.0. It doesn't happen on 0.8.6. Is there >> > any >> > thing I can do to check? >> > thanks >> > Ramesh >> > >> > On Mon, Oct 3, 2011 at 5:15 PM, Jonathan Ellis <jbel...@gmail.com> >> > wrote: >> >> >> >> My suspicion would be that it has more to do with "rare case when >> >> running with 5000 CFs" than "1.0 regression." >> >> >> >> On Mon, Oct 3, 2011 at 5:00 PM, Ramesh Natarajan <rames...@gmail.com> >> >> wrote: >> >> > We have about 5000 column family and when we run the nodetool cfstats >> >> > it >> >> > throws out this exception... this is running 1.0.0-rc1 >> >> > This seems to work on 0.8.6. Is this a bug in 1.0.0? >> >> > >> >> > thanks >> >> > Ramesh >> >> > Keyspace: system >> >> > Read Count: 28 >> >> > Read Latency: 5.8675 ms. >> >> > Write Count: 3 >> >> > Write Latency: 0.166 ms. >> >> > Pending Tasks: 0 >> >> > Column Family: Schema >> >> > SSTable count: 4 >> >> > Space used (live): 4293758276 >> >> > Space used (total): 4293758276 >> >> > Number of Keys (estimate): 5376 >> >> > Memtable Columns Count: 0 >> >> > Memtable Data Size: 0 >> >> > Memtable Switch Count: 0 >> >> > Read Count: 3 >> >> > Read Latency: NaN ms. >> >> > Write Count: 0 >> >> > Write Latency: NaN ms. >> >> > Pending Tasks: 0 >> >> > Key cache capacity: 53 >> >> > Key cache size: 2 >> >> > Key cache hit rate: NaN >> >> > Row cache: disabled >> >> > Compacted row minimum size: 104 >> >> > Compacted row maximum size: 1955666 >> >> > Compacted row mean size: 1508515 >> >> > Column Family: HintsColumnFamily >> >> > SSTable count: 0 >> >> > Space used (live): 0 >> >> > Space used (total): 0 >> >> > Number of Keys (estimate): 0 >> >> > Memtable Columns Count: 0 >> >> > Memtable Data Size: 0 >> >> > Memtable Switch Count: 0 >> >> > Read Count: 5 >> >> > Read Latency: NaN ms. >> >> > Write Count: 0 >> >> > Write Latency: NaN ms. >> >> > Pending Tasks: 0 >> >> > Key cache capacity: 1 >> >> > Key cache size: 0 >> >> > Key cache hit rate: NaN >> >> > Row cache: disabled >> >> > Compacted row minimum size: 0 >> >> > Compacted row maximum size: 0 >> >> > Compacted row mean size: 0 >> >> > Column Family: LocationInfo >> >> > SSTable count: 1 >> >> > Space used (live): 6947 >> >> > Space used (total): 6947 >> >> > Number of Keys (estimate): 128 >> >> > Memtable Columns Count: 0 >> >> > Memtable Data Size: 0 >> >> > Memtable Switch Count: 2 >> >> > Read Count: 20 >> >> > Read Latency: NaN ms. >> >> > Write Count: 3 >> >> > Write Latency: NaN ms. >> >> > Pending Tasks: 0 >> >> > Key cache capacity: 1 >> >> > Key cache size: 1 >> >> > Key cache hit rate: NaN >> >> > Row cache: disabled >> >> > Compacted row minimum size: 73 >> >> > Compacted row maximum size: 258 >> >> > Compacted row mean size: 185 >> >> > Column Family: Migrations >> >> > SSTable count: 4 >> >> > Space used (live): 4315909643 >> >> > Space used (total): 4315909643 >> >> > Number of Keys (estimate): 512 >> >> > Memtable Columns Count: 0 >> >> > Memtable Data Size: 0 >> >> > Memtable Switch Count: 0 >> >> > Read Count: 0 >> >> > Read Latency: NaN ms. >> >> > Write Count: 0 >> >> > Write Latency: NaN ms. >> >> > Pending Tasks: 0 >> >> > Key cache capacity: 5 >> >> > Key cache size: 0 >> >> > Key cache hit rate: NaN >> >> > Row cache: disabled >> >> > Compacted row minimum size: 5839589 >> >> > Compacted row maximum size: 9223372036854775807 >> >> > Exception in thread "main" java.lang.IllegalStateException: Unable to >> >> > compute ceiling for max when histogram overflowed >> >> > at >> >> > >> >> > >> >> > org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:170) >> >> > at >> >> > >> >> > org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395) >> >> > at >> >> > >> >> > >> >> > org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStore.java:275) >> >> > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown >> >> > Source) >> >> > at >> >> > >> >> > >> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> > at java.lang.reflect.Method.invoke(Method.java:597) >> >> > at >> >> > >> >> > >> >> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) >> >> > at >> >> > >> >> > >> >> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) >> >> > at >> >> > >> >> > >> >> > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) >> >> > at >> >> > >> >> > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) >> >> > at >> >> > >> >> > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) >> >> > at >> >> > >> >> > >> >> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) >> >> > at >> >> > >> >> > >> >> > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) >> >> > at >> >> > >> >> > >> >> > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) >> >> > at >> >> > >> >> > >> >> > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) >> >> > at >> >> > >> >> > >> >> > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) >> >> > at >> >> > >> >> > >> >> > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) >> >> > at >> >> > >> >> > >> >> > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) >> >> > at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown >> >> > Source) >> >> > at >> >> > >> >> > >> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> > at java.lang.reflect.Method.invoke(Method.java:597) >> >> > at >> >> > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >> >> > at sun.rmi.transport.Transport$1.run(Transport.java:159) >> >> > at java.security.AccessController.doPrivileged(Native Method) >> >> > at >> >> > sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> >> > at >> >> > >> >> > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >> >> > at >> >> > >> >> > >> >> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> >> > at >> >> > >> >> > >> >> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> >> > at >> >> > >> >> > >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> > at >> >> > >> >> > >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> > at java.lang.Thread.run(Thread.java:662) >> >> > >> >> >> >> >> >> >> >> -- >> >> Jonathan Ellis >> >> Project Chair, Apache Cassandra >> >> co-founder of DataStax, the source for professional Cassandra support >> >> http://www.datastax.com >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com > > >
