nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small keyspace with no compression, about 250MB. The rest taken by the second keyspace with leveled compaction and snappy compressed.
The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM. On Thu, Jun 7, 2012 at 2:52 AM, aaron morton <aa...@thelastpickle.com>wrote: > How much data do you have on the node ? > Was this a previously running system that was upgraded ? > > > with disk_access_mode mmap_index_only and mmap I see OOM map failed > error on SSTableBatchOpen thread > Do you have the stack trace from the log ? > > > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:6,1,main] > > java.lang.StackOverflowError > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > Was there more to this stack trace ? > What were the log messages before this error ? > > > > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line > 122) Heap size: 1525415936/1525415936 > The JVM only has 1.5 G of ram, this is at the lower limit. If you have > some data to load I would not be surprised if it failed to start. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 7/06/2012, at 8:41 AM, Javier Sotelo wrote: > > > Hi All, > > > > On SuSe Linux blade with 6GB of RAM. > > > > with disk_access_mode mmap_index_only and mmap I see OOM map failed > error on SSTableBatchOpen thread. cat /proc/<pid>/maps shows a peak of > 53521 right before it dies. vm.max_map_count = 1966080 and > /proc/<pid>/limits shows unlimited locked memory. > > > > with disk_access_mode standard, the node does start up but I see the > repeated error: > > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:6,1,main] > > java.lang.StackOverflowError > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > ... > > > > I'm not sure the second error is related to the first. I prefer to run > with full mmap but I have run out of ideas. Is there anything else I can do > to debug this? > > > > Here's startup settings from debug log: > > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line > 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 > > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line > 122) Heap size: 1525415936/1525415936 > > ... > > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA > mlockall successful > > ... > > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191) > DiskAccessMode is standard, indexAccessMode is standard > > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247) > Global memtable threshold is enabled at 484MB > > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96) > Initializing key cache with capacity of 72 MBs. > > INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107) > Scheduling key cache save to each 14400 seconds (going to save all keys). > > INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121) > Initializing row cache with capacity of 0 MBs and provider > org.apache.cassandra.cache.SerializingCacheProvider > > INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133) > Scheduling row cache save to each 0 seconds (going to save all keys). > > > > Thanks In Advance, > > Javier > >
