On Mon, Jan 17, 2011 at 1:20 PM, Karl Hiramoto <k...@hiramoto.org> wrote: > On 01/17/11 15:54, Edward Capriolo wrote: >> Just to head the next possible problem. If you run 'nodetool cleanup' >> on each node and some of your nodes still have more data then others, >> then it probably means your are writing the majority of data to a few >> keys. ( you probably do not want to do that ) >> >> If that happens, you can use nodetool cfstats on each node and ensure >> that the 'max row compacted size' is roughly the same on all nodes. If >> you have one or two really big rows that could explain your imbalance. > > > Well, I did a lengthy repair/cleanup on each node. but still have data > mainly on two nodes (I have RF=2) > ./apache-cassandra-0.7.0/bin/nodetool --host host3 ring > Address Status State Load Owns > Token > > 119098828422328462212181112601118874007 > 10.1.4.10 Up Normal 347.11 MB 30.00% > 0 > 10.1.4.12 Up Normal 49.41 KB 20.00% > 34028236692093846346337460743176821145 > 10.1.4.13 Up Normal 54.35 KB 20.00% > 68056473384187692692674921486353642290 > 10.1.4.15 Up Normal 19.09 GB 16.21% > 95643579558861158157614918209686336369 > 10.1.4.14 Up Normal 15.62 GB 13.79% > 119098828422328462212181112601118874007 > > > in "cfstats" i see: > Compacted row minimum size: 1131752 > Compacted row maximum size: 8582860529 > Compacted row mean size: 1402350749 > > on the lowest used node i see: > Compacted row minimum size: 0 > Compacted row maximum size: 0 > Compacted row mean size: 0 > > I basicly have MyKeyspace.Offer[UID] = value my "value" is at most > 500 bytes. > > UID i just use a 12 random alpha numeric values [A-Z][0-9] > > Should i try and adjust my tokens to fix the imbalance or something else? > > I'm using Redhat EL 5.5 > > java -version > java version "1.6.0_17" > OpenJDK Runtime Environment (IcedTea6 1.7.5) (rhel-1.16.b17.el5-x86_64) > OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) > > I have some errors in my logs: > > ERROR [ReadStage:1747] 2011-01-17 18:13:53,988 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.AssertionError > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:178) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:132) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:70) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1215) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077) > at org.apache.cassandra.db.Table.getRow(Table.java:384) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > ERROR [ReadStage:1747] 2011-01-17 18:13:53,989 > AbstractCassandraDaemon.java (line 91) Fatal exception in thread > Thread[ReadStage:1747,5,main] > java.lang.AssertionError > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:178) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:132) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:70) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1215) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077) > at org.apache.cassandra.db.Table.getRow(Table.java:384) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > > > Thanks, > > Karl > >
cfstats is reporting you have an 8GB Row! I think you could be writing all your data to a few keys.
