Hi All,
I applied Cassandra patch for issue 8248 to see what it do.
Now I noticed below errors in my system.log :
ERROR [NonPeriodicTasks:1] 2015-01-07 10:55:48,869 CassandraDaemon.java:153
- Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
ERROR [NonPeriodicTasks:1] 2015-01-07 10:55:48,870 CassandraDaemon.java:153
- Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:7] 2015-01-07 10:55:48,870
ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress:
164 (0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:3] 2015-01-07 10:55:48,871 Memtable.java:325 -
Writing Memtable-compactions_in_progress@1529300710(0 serialized bytes, 1
ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:3] 2015-01-07 10:55:48,878 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1327-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1420608014522,
position=44984450)
INFO [CompactionExecutor:8] 2015-01-07 10:55:48,879
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1325-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1324-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1326-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1327-Data.db')]
ERROR [CompactionExecutor:7] 2015-01-07 10:55:48,880
CassandraDaemon.java:153 - Exception in thread
Thread[CompactionExecutor:7,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
*Please help in knowing the reason for these errors.*
regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 10:46 PM, Ryan Svihla <r...@foundev.pro> wrote:
> That even with that patch you'll likely run heap pressure with batches of
> that size, so either increase your heap and take the GC hit on CPU (and
> have longer GCs) or don't use large batches.
>
> The batch conversation is a bigger one which I discuss here
> http://lostechies.com/ryansvihla/2014/08/28/cassandra-batch-loading-without-the-batch-keyword/
>
> On Tue, Jan 6, 2015 at 10:07 AM, Rahul Bhardwaj <
> rahul.bhard...@indiamart.com> wrote:
>
>> Thanks Ryan... We will keep ur valuable suggestion in resolving this
>> issue.. But what is your take on Cassandra patch for issue 8248 to resolve
>> this.
>>
>>
>> On Tuesday, January 6, 2015, Ryan Svihla <r...@foundev.pro> wrote:
>>
>>> Btw side note here, you're using GIANT Batches, and the logs are
>>> indicating such, this will cause a signficant amount of heap pressure.
>>>
>>> The root cause fix is not to use giant batches in the first place.
>>>
>>> On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj <
>>> rahul.bhard...@indiamart.com> wrote:
>>>
>>>> Hi Joe..
>>>>
>>>> Thanks for your valuable solution.. it worked.
>>>>
>>>> But for this problem
>>>>
>>>> *The processes are killed by kernel, coz they are eating all memory
>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>> because we have 64 GB RAM.*
>>>>
>>>> Should I apply patch given for issue
>>>> https://issues.apache.org/jira/browse/CASSANDRA-8248 ??
>>>>
>>>>
>>>> regards:
>>>> Rahul Bhardwaj
>>>>
>>>> On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <joe.ram...@mac.com> wrote:
>>>>
>>>>> Thanks Rahul and good luck! I’m really curious to hear what the
>>>>> result is.
>>>>>
>>>>>
>>>>> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <
>>>>> rahul.bhard...@indiamart.com> wrote:
>>>>>
>>>>> Thanks for your response.. i will get back to you with my findings.
>>>>>
>>>>> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <joe.ram...@mac.com>
>>>>> wrote:
>>>>>
>>>>>> That should be “writing too many bytes” not “waiting too many bytes”
>>>>>> just for clarity’s sake.
>>>>>>
>>>>>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <joe.ram...@mac.com> wrote:
>>>>>>
>>>>>> I’m not an expert. Really just learning this myself but it looks
>>>>>> like according to the stack you’re getting an exception waiting too many
>>>>>> bytes to the commit log.
>>>>>>
>>>>>> That’s controlled by commit_log_segment_size_in_mb setting. The
>>>>>> maximum write size that C* will allow is half of the value set for this
>>>>>> parameter so if it’s set for 32 (default) that means your max write would
>>>>>> be 16MB. (And that’s what’s getting reported in the stack trace. You’re
>>>>>> writing 16965030 bytes (16.18M) and your max write is 16MB so it’s
>>>>>> throwing
>>>>>> the exception. Try setting commit_log_segment_size_in_mb from 32 to 64
>>>>>> and
>>>>>> see if the error goes away. It really should get tuned for the amount of
>>>>>> writes but that’ll tell you at least if this is the issue.
>>>>>>
>>>>>> Let me know how it goes!
>>>>>>
>>>>>> Joe
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <
>>>>>> rahul.bhard...@indiamart.com> wrote:
>>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> PFB output of system.log
>>>>>>
>>>>>> tail -n 100 system.log
>>>>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
>>>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>>>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
>>>>>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
>>>>>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition
>>>>>> merge
>>>>>> counts were {1:246186, 2:11302, 3:2259, 4:538, }
>>>>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
>>>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>>>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
>>>>>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
>>>>>> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>>>>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 433053, exceeding specified threshold of 5120 by 427933.
>>>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>>>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms =
>>>>>> 39.871073MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 433819, exceeding specified threshold of 5120 by 428699.
>>>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>>>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms =
>>>>>> 41.097876MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 436755, exceeding specified threshold of 5120 by 431635.
>>>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>>>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms =
>>>>>> 42.402570MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on
>>>>>> thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
>>>>>> large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>>>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms =
>>>>>> 37.102249MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
>>>>>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden
>>>>>> Space:
>>>>>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>>>>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for
>>>>>> [clickstream.im_csl_glusr_daily_activity_count] is of size 60021,
>>>>>> exceeding
>>>>>> specified threshold of 5120 by 54901.
>>>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>>>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms =
>>>>>> 36.593999MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 517769, exceeding specified threshold of 5120 by 512649.
>>>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>>>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms =
>>>>>> 38.715390MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>>> 13220136, exceeding specified threshold of 5120 by 13215016.
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on
>>>>>> thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>>>> Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887
>>>>>> bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> ... 4 common frames omitted
>>>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>>>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms =
>>>>>> 41.740030MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>>> 13136950, exceeding specified threshold of 5120 by 13131830.
>>>>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129
>>>>>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980
>>>>>> (24%)
>>>>>> on-heap, 0 (0%) off-heap
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131
>>>>>> Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505
>>>>>> serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
>>>>>> (21516225 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
>>>>>> (361277 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
>>>>>> (543861 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
>>>>>> (481902 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
>>>>>> (460053 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
>>>>>> (395084 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>>>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms =
>>>>>> 39.284734MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>>>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms =
>>>>>> 37.102249MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>>> 13480721, exceeding specified threshold of 5120 by 13475601.
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on
>>>>>> thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>>>> Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032
>>>>>> bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> ... 4 common frames omitted
>>>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>>>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms =
>>>>>> 44.522699MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>>>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms =
>>>>>> 40.475181MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on
>>>>>> thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
>>>>>> large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards:
>>>>>> Rahul Bhardwaj
>>>>>>
>>>>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
>>>>>> rahul.bhard...@indiamart.com> wrote:
>>>>>>
>>>>>>> Hi Joe,
>>>>>>>
>>>>>>> PFA heap dump
>>>>>>>
>>>>>>>
>>>>>>> regards:
>>>>>>> Rahul Bhardwaj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ram...@mac.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Did you try generating a heap dump so you can look through it to
>>>>>>>> see what’s actually happened?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>>>>>>> rahul.bhard...@indiamart.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We are using cassandra 2.1 version in a cluster of three machines
>>>>>>>> each with 64 GB RAM
>>>>>>>>
>>>>>>>> The processes are killed by kernel, coz they are eating all memory
>>>>>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>>>>>> because we have 64 GB RAM.
>>>>>>>>
>>>>>>>> Please help.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards:
>>>>>>>> Rahul Bhardwaj
>>>>>>>>
>>>>>>>>
>>>>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest
>>>>>>>> updates on this and more: <https://plus.google.com/+indiamart>
>>>>>>>> <https://www.facebook.com/IndiaMART>
>>>>>>>> <https://twitter.com/IndiaMART> Mobile Channel:
>>>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>>>> <http://m.indiamart.com/>
>>>>>>>>
>>>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>>>>> kyunki Kaam Yahin Banta Hai
>>>>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>> Mobile Channel:
>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>> <http://m.indiamart.com/>
>>>>>>
>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>>> kyunki Kaam Yahin Banta Hai
>>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>> kyunki Kaam Yahin Banta Hai
>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>> kyunki Kaam Yahin Banta Hai
>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Ryan Svihla
>>>
>>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>
>
>
> --
>
> Thanks,
> Ryan Svihla
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
