Different node same hardware now gets the stack overflow error but I found
the part of the stack trace that is more interesting:
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators.size(Iterators.java:129)
at com.google.common.collect.Sets$3.size(Sets.java:670)
at com.google.common.collect.Iterables.size(Iterables.java:80)
at
org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557)
at
org.apache.cassandra.db.compaction.CompactionController.<init>(CompactionController.java:79)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105)
at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
Is it time for a JIRA ticket?
On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo <javier.a.sot...@gmail.com>wrote:
> nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small
> keyspace with no compression, about 250MB. The rest taken by the second
> keyspace with leveled compaction and snappy compressed.
>
> The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM.
>
>
> On Thu, Jun 7, 2012 at 2:52 AM, aaron morton <aa...@thelastpickle.com>wrote:
>
>> How much data do you have on the node ?
>> Was this a previously running system that was upgraded ?
>>
>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>> error on SSTableBatchOpen thread
>> Do you have the stack trace from the log ?
>>
>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[CompactionExecutor:6,1,main]
>> > java.lang.StackOverflowError
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> Was there more to this stack trace ?
>> What were the log messages before this error ?
>>
>>
>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line
>> 122) Heap size: 1525415936/1525415936
>> The JVM only has 1.5 G of ram, this is at the lower limit. If you have
>> some data to load I would not be surprised if it failed to start.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote:
>>
>> > Hi All,
>> >
>> > On SuSe Linux blade with 6GB of RAM.
>> >
>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>> error on SSTableBatchOpen thread. cat /proc/<pid>/maps shows a peak of
>> 53521 right before it dies. vm.max_map_count = 1966080 and
>> /proc/<pid>/limits shows unlimited locked memory.
>> >
>> > with disk_access_mode standard, the node does start up but I see the
>> repeated error:
>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[CompactionExecutor:6,1,main]
>> > java.lang.StackOverflowError
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>> > ...
>> >
>> > I'm not sure the second error is related to the first. I prefer to run
>> with full mmap but I have run out of ideas. Is there anything else I can do
>> to debug this?
>> >
>> > Here's startup settings from debug log:
>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line
>> 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31
>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line
>> 122) Heap size: 1525415936/1525415936
>> > ...
>> > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA
>> mlockall successful
>> > ...
>> > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191)
>> DiskAccessMode is standard, indexAccessMode is standard
>> > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247)
>> Global memtable threshold is enabled at 484MB
>> > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96)
>> Initializing key cache with capacity of 72 MBs.
>> > INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107)
>> Scheduling key cache save to each 14400 seconds (going to save all keys).
>> > INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121)
>> Initializing row cache with capacity of 0 MBs and provider
>> org.apache.cassandra.cache.SerializingCacheProvider
>> > INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133)
>> Scheduling row cache save to each 0 seconds (going to save all keys).
>> >
>> > Thanks In Advance,
>> > Javier
>>
>>
>
