I've filed https://issues.apache.org/jira/browse/CASSANDRA-3957 as a bug. Any further light you can shed here would be useful. (Is row cache enabled? Is JNA installed?)
On Mon, Feb 20, 2012 at 5:43 AM, Thomas van Neerijnen <t...@bossastudios.com> wrote: > Hi all > > I am running the Apache packaged Cassandra 1.0.7 on Ubuntu 11.10. > It has been running fine for over a month however I encountered the below > error yesterday which almost immediately resulted in heap usage rising > quickly to almost 100% and client requests timing out on the affected node. > I gave up waiting for the init script to stop Cassandra and killed it myself > after about 3 minutes, restarted it and it has been fine since. Anyone seen > this before? > > Here is the error in the output.log: > > ERROR 10:51:44,282 Fatal exception in thread > Thread[COMMIT-LOG-WRITER,5,main] > java.lang.AssertionError: Final buffer length 4690 to accomodate data size > of 2347 (predicted 2344) for RowMutation(keyspace='Player', > key='36336138643338652d366162302d343334392d383466302d356166643863353133356465', > modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019 > []),SuperColumn(owneditem_1024 []),SuperColumn(owneditem_1026 > []),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077 > []),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094 > []),SuperColumn(owneditem_1130 []),SuperColumn(owneditem_1136 > []),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142 > []),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218 > [636f6e6e6563746564:false:5@1329648704269002,63757272656e744865616c7468:false:3@1329648704269006,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001,6c61737444657374726f79656454696d65:false:1@1329648704269008,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003,79:false:3@1329648704269004,]),SuperColumn(owneditem_133 > []),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135 > []),SuperColumn(owneditem_141 []),SuperColumn(owneditem_147 > []),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159 > []),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253 > []),SuperColumn(owneditem_422 []),SuperColumn(owneditem_438 > []),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521 > []),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525 > []),SuperColumn(owneditem_562 []),SuperColumn(owneditem_61 > []),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636 > []),SuperColumn(owneditem_71 []),SuperColumn(owneditem_712 > []),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728 > []),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797 > []),SuperColumn(owneditem_798 []),SuperColumn(owneditem_838 > []),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847 > []),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851 > []),SuperColumn(owneditem_852 []),SuperColumn(owneditem_853 > []),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857 > []),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874 > []),SuperColumn(owneditem_884 []),SuperColumn(owneditem_886 > []),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91 > []),SuperColumn(owneditem_911 []),SuperColumn(owneditem_930 > []),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937 > []),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945 > []),SuperColumn(owneditem_962 []),SuperColumn(owneditem_963 > []),SuperColumn(owneditem_964 []),])]) > at > org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682) > at > org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122) > at > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:662) > WARN 10:51:54,302 Heap is 0.764063958911146 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN 10:51:54,303 Flushing CFS(Keyspace='Player', > ColumnFamily='PlayerDetail') to relieve memory pressure > INFO 11:00:41,162 Started hinted handoff for token: > 121529416757478022665490931225631504090 with IP: /10.16.96.212 > INFO 11:00:41,163 Finished hinted handoff of 0 rows to endpoint > /10.16.96.212 > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor192] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor165] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor202] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor232] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor146] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor181] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor190] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor214] > ... > [Unloading class sun.reflect.GeneratedConstructorAccessor19] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor209] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor227] > [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor205] > [Unloading class sun.reflect.GeneratedMethodAccessor202] > > The "Unloading" messages continued, with occasional mentions of hinted > handoffs in between, until I killed Cassandra. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
