Btw side note here, you're using GIANT Batches, and the logs are indicating such, this will cause a signficant amount of heap pressure.
The root cause fix is not to use giant batches in the first place. On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj <rahul.bhard...@indiamart.com > wrote: > Hi Joe.. > > Thanks for your valuable solution.. it worked. > > But for this problem > > *The processes are killed by kernel, coz they are eating all memory > (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) > because we have 64 GB RAM.* > > Should I apply patch given for issue > https://issues.apache.org/jira/browse/CASSANDRA-8248 ?? > > > regards: > Rahul Bhardwaj > > On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <joe.ram...@mac.com> wrote: > >> Thanks Rahul and good luck! I’m really curious to hear what the result >> is. >> >> >> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <rahul.bhard...@indiamart.com> >> wrote: >> >> Thanks for your response.. i will get back to you with my findings. >> >> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <joe.ram...@mac.com> wrote: >> >>> That should be “writing too many bytes” not “waiting too many bytes” >>> just for clarity’s sake. >>> >>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <joe.ram...@mac.com> wrote: >>> >>> I’m not an expert. Really just learning this myself but it looks like >>> according to the stack you’re getting an exception waiting too many bytes >>> to the commit log. >>> >>> That’s controlled by commit_log_segment_size_in_mb setting. The maximum >>> write size that C* will allow is half of the value set for this parameter >>> so if it’s set for 32 (default) that means your max write would be 16MB. >>> (And that’s what’s getting reported in the stack trace. You’re writing >>> 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the >>> exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see >>> if the error goes away. It really should get tuned for the amount of >>> writes but that’ll tell you at least if this is the issue. >>> >>> Let me know how it goes! >>> >>> Joe >>> >>> >>> >>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <rahul.bhard...@indiamart.com> >>> wrote: >>> >>> Hi Joe, >>> >>> PFB output of system.log >>> >>> tail -n 100 system.log >>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 >>> CompactionTask.java:251 - Compacted 4 sstables to >>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. >>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = >>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge >>> counts were {1:246186, 2:11302, 3:2259, 4:538, } >>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 >>> CompactionTask.java:251 - Compacted 4 sstables to >>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. >>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total >>> partitions merged to 1. Partition merge counts were {1:1, 2:3, } >>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - >>> Batch of prepared statements for [clickstream.business_feed] is of size >>> 433053, exceeding specified threshold of 5120 by 427933. >>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')] >>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch >>> of prepared statements for [clickstream.business_feed] is of size 433819, >>> exceeding specified threshold of 5120 by 428699. >>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')] >>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch >>> of prepared statements for [clickstream.business_feed] is of size 436755, >>> exceeding specified threshold of 5120 by 431635. >>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')] >>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 >>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>> Thread[SharedPool-Worker-1,5,main]: {} >>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too >>> large for the maxiumum size of 16777216 >>> at >>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) ~[na:1.7.0_71] >>> at >>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')] >>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - >>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: >>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936 >>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch >>> of prepared statements for [clickstream.im_csl_glusr_daily_activity_count] >>> is of size 60021, exceeding specified threshold of 5120 by 54901. >>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')] >>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch >>> of prepared statements for [clickstream.business_feed] is of size 517769, >>> exceeding specified threshold of 5120 by 512649. >>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')] >>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - >>> Batch of prepared statements for [clickstream.im_csl_log] is of size >>> 13220136, exceeding specified threshold of 5120 by 13215016. >>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 >>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>> Thread[SharedPool-Worker-1,5,main]: {} >>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation >>> of 16800887 bytes is too large for the maxiumum size of 16777216 >>> at >>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) ~[na:1.7.0_71] >>> at >>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 >>> bytes is too large for the maxiumum size of 16777216 >>> at >>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> ... 4 common frames omitted >>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')] >>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch >>> of prepared statements for [clickstream.im_csl_log] is of size 13136950, >>> exceeding specified threshold of 5120 by 13131830. >>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 >>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) >>> on-heap, 0 (0%) off-heap >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 >>> - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, >>> 1530359 ops, 20%/0% of on/off-heap limit) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 >>> - Completed flushing >>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db >>> (21516225 bytes) for commitlog position >>> ReplayPosition(segmentId=1420516895898, position=16708232) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 >>> - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 >>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 >>> - Completed flushing >>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db >>> (361277 bytes) for commitlog position >>> ReplayPosition(segmentId=1420516895898, position=16708232) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 >>> - Writing >>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 >>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 >>> - Completed flushing >>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db >>> (543861 bytes) for commitlog position >>> ReplayPosition(segmentId=1420516895898, position=16708232) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 >>> - Writing >>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 >>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 >>> - Completed flushing >>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db >>> (481902 bytes) for commitlog position >>> ReplayPosition(segmentId=1420516895898, position=16708232) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 >>> - Writing >>> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 >>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 >>> - Completed flushing >>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db >>> (460053 bytes) for commitlog position >>> ReplayPosition(segmentId=1420516895898, position=16708232) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 >>> - Writing >>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 >>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 >>> - Completed flushing >>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db >>> (395084 bytes) for commitlog position >>> ReplayPosition(segmentId=1420516895898, position=16708232) >>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')] >>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')] >>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch >>> of prepared statements for [clickstream.im_csl_log] is of size 13480721, >>> exceeding specified threshold of 5120 by 13475601. >>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 >>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>> Thread[SharedPool-Worker-1,5,main]: {} >>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation >>> of 17138032 bytes is too large for the maxiumum size of 16777216 >>> at >>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) ~[na:1.7.0_71] >>> at >>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 >>> bytes is too large for the maxiumum size of 16777216 >>> at >>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> ... 4 common frames omitted >>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')] >>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 >>> CompactionTask.java:136 - Compacting >>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')] >>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 >>> CompactionTask.java:251 - Compacted 1 sstables to >>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. >>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. >>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 >>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>> Thread[SharedPool-Worker-1,5,main]: {} >>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too >>> large for the maxiumum size of 16777216 >>> at >>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) ~[na:1.7.0_71] >>> at >>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>> >>> >>> >>> Regards: >>> Rahul Bhardwaj >>> >>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj < >>> rahul.bhard...@indiamart.com> wrote: >>> >>>> Hi Joe, >>>> >>>> PFA heap dump >>>> >>>> >>>> regards: >>>> Rahul Bhardwaj >>>> >>>> >>>> >>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ram...@mac.com> wrote: >>>> >>>>> Did you try generating a heap dump so you can look through it to see >>>>> what’s actually happened? >>>>> >>>>> >>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj < >>>>> rahul.bhard...@indiamart.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> We are using cassandra 2.1 version in a cluster of three machines each >>>>> with 64 GB RAM >>>>> >>>>> The processes are killed by kernel, coz they are eating all memory >>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) >>>>> because we have 64 GB RAM. >>>>> >>>>> Please help. >>>>> >>>>> >>>>> Regards: >>>>> Rahul Bhardwaj >>>>> >>>>> >>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates >>>>> on this and more: <https://plus.google.com/+indiamart> >>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>>>> Mobile Channel: >>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>>>> <http://m.indiamart.com/> >>>>> >>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, >>>>> kyunki Kaam Yahin Banta Hai >>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >>>>> >>>>> >>>>> >>>> >>> >>> >>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on >>> this and more: <https://plus.google.com/+indiamart> >>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>> Mobile Channel: >>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>> <http://m.indiamart.com/> >>> >>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki >>> Kaam >>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >>> >>> >>> >>> >> >> >> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on >> this and more: <https://plus.google.com/+indiamart> >> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >> Mobile Channel: >> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >> <https://play.google.com/store/apps/details?id=com.indiamart.m> >> <http://m.indiamart.com/> >> >> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam >> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >> >> >> > > > Follow IndiaMART.com <http://www.indiamart.com> for latest updates on > this and more: <https://plus.google.com/+indiamart> > <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> > Mobile Channel: > <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> > <https://play.google.com/store/apps/details?id=com.indiamart.m> > <http://m.indiamart.com/> > > <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> > Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam > Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! > -- Thanks, Ryan Svihla
