Hi all
I am running the Apache packaged Cassandra 1.0.7 on Ubuntu 11.10.
It has been running fine for over a month however I encountered the below
error yesterday which almost immediately resulted in heap usage rising
quickly to almost 100% and client requests timing out on the affected node.
I gave up waiting for the init script to stop Cassandra and killed it
myself after about 3 minutes, restarted it and it has been fine since.
Anyone seen this before?
Here is the error in the output.log:
ERROR 10:51:44,282 Fatal exception in thread
Thread[COMMIT-LOG-WRITER,5,main]
java.lang.AssertionError: Final buffer length 4690 to accomodate data size
of 2347 (predicted 2344) for RowMutation(keyspace='Player',
key='36336138643338652d366162302d343334392d383466302d356166643863353133356465',
modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019
[]),SuperColumn(owneditem_1024 []),SuperColumn(owneditem_1026
[]),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077
[]),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094
[]),SuperColumn(owneditem_1130 []),SuperColumn(owneditem_1136
[]),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142
[]),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218
[636f6e6e6563746564:false:5@1329648704269002
,63757272656e744865616c7468:false:3@1329648704269006
,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007
,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001
,6c61737444657374726f79656454696d65:false:1@1329648704269008
,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005
,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003
,79:false:3@1329648704269004,]),SuperColumn(owneditem_133
[]),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135
[]),SuperColumn(owneditem_141 []),SuperColumn(owneditem_147
[]),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159
[]),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253
[]),SuperColumn(owneditem_422 []),SuperColumn(owneditem_438
[]),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521
[]),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525
[]),SuperColumn(owneditem_562 []),SuperColumn(owneditem_61
[]),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636
[]),SuperColumn(owneditem_71 []),SuperColumn(owneditem_712
[]),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728
[]),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797
[]),SuperColumn(owneditem_798 []),SuperColumn(owneditem_838
[]),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847
[]),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851
[]),SuperColumn(owneditem_852 []),SuperColumn(owneditem_853
[]),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857
[]),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874
[]),SuperColumn(owneditem_884 []),SuperColumn(owneditem_886
[]),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91
[]),SuperColumn(owneditem_911 []),SuperColumn(owneditem_930
[]),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937
[]),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945
[]),SuperColumn(owneditem_962 []),SuperColumn(owneditem_963
[]),SuperColumn(owneditem_964 []),])])
at
org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
at
org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
at
org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599)
at
org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
WARN 10:51:54,302 Heap is 0.764063958911146 full. You may need to reduce
memtable and/or cache sizes. Cassandra will now flush up to the two
largest memtables to free up memory. Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
WARN 10:51:54,303 Flushing CFS(Keyspace='Player',
ColumnFamily='PlayerDetail') to relieve memory pressure
INFO 11:00:41,162 Started hinted handoff for token:
121529416757478022665490931225631504090 with IP: /10.16.96.212
INFO 11:00:41,163 Finished hinted handoff of 0 rows to endpoint /
10.16.96.212
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor192]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor165]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor202]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor232]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor146]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor181]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor190]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor214]
...
[Unloading class sun.reflect.GeneratedConstructorAccessor19]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor209]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor227]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor205]
[Unloading class sun.reflect.GeneratedMethodAccessor202]
The "Unloading" messages continued, with occasional mentions of hinted
handoffs in between, until I killed Cassandra.
