I’m not an expert. Really just learning this myself but it looks like
according to the stack you’re getting an exception waiting too many bytes to
the commit log.
That’s controlled by commit_log_segment_size_in_mb setting. The maximum write
size that C* will allow is half of the value set for this parameter so if it’s
set for 32 (default) that means your max write would be 16MB. (And that’s
what’s getting reported in the stack trace. You’re writing 16965030 bytes
(16.18M) and your max write is 16MB so it’s throwing the exception. Try
setting commit_log_segment_size_in_mb from 32 to 64 and see if the error goes
away. It really should get tuned for the amount of writes but that’ll tell you
at least if this is the issue.
Let me know how it goes!
Joe
> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <rahul.bhard...@indiamart.com>
> wrote:
>
> Hi Joe,
>
> PFB output of system.log
>
> tail -n 100 system.log
> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
> CompactionTask.java:251 - Compacted 4 sstables to
> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
> counts were {1:246186, 2:11302, 3:2259, 4:538, }
> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
> CompactionTask.java:251 - Compacted 4 sstables to
> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.business_feed] is of size 433053,
> exceeding specified threshold of 5120 by 427933.
> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.business_feed] is of size 433819,
> exceeding specified threshold of 5120 by 428699.
> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.business_feed] is of size 436755,
> exceeding specified threshold of 5120 by 431635.
> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large
> for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew
> GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is of
> size 60021, exceeding specified threshold of 5120 by 54901.
> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.business_feed] is of size 517769,
> exceeding specified threshold of 5120 by 512649.
> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.im_csl_log] is of size 13220136,
> exceeding specified threshold of 5120 by 13215016.
> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of
> 16800887 bytes is too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is
> too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> ... 4 common frames omitted
> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.im_csl_log] is of size 13136950,
> exceeding specified threshold of 5120 by 13131830.
> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 -
> Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 -
> Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359
> ops, 20%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
> (21516225 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 -
> Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized
> bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
> (361277 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 -
> Writing
> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
> (543861 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 -
> Writing
> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
> (481902 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 -
> Writing
> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
> (460053 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 -
> Writing
> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
> (395084 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of
> prepared statements for [clickstream.im_csl_log] is of size 13480721,
> exceeding specified threshold of 5120 by 13475601.
> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of
> 17138032 bytes is too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is
> too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> ... 4 common frames omitted
> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large
> for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>
>
>
> Regards:
> Rahul Bhardwaj
>
> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <rahul.bhard...@indiamart.com
> <mailto:rahul.bhard...@indiamart.com>> wrote:
> Hi Joe,
>
> PFA heap dump
>
>
> regards:
> Rahul Bhardwaj
>
>
>
> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ram...@mac.com
> <mailto:joe.ram...@mac.com>> wrote:
> Did you try generating a heap dump so you can look through it to see what’s
> actually happened?
>
>
>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <rahul.bhard...@indiamart.com
>> <mailto:rahul.bhard...@indiamart.com>> wrote:
>>
>> Hi,
>>
>> We are using cassandra 2.1 version in a cluster of three machines each with
>> 64 GB RAM
>>
>> The processes are killed by kernel, coz they are eating all memory
>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because
>> we have 64 GB RAM.
>>
>> Please help.
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this
>> and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
>> Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki
>> Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
>
>
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this
> and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
> Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
