Check out that the limits here are set correctly: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html
particularly the: * mmap limit which is really what this looks like... * nproc limit which on some distros defaults to 1024 can be your issue (maximum number of threads).. i don't think this is it but maybe. If you think you have it all correctly but still hitting limit verify that the process is picking them up: cat /proc/`cat /var/run/cassandra/cassandra.pid`/limits or cat /proc/whateverCassandraPIDIs/limits should look (something) like: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 100000 100000 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 16382 16382 signals Max msgqueue size 819200 819200 bytes Max nice priority 20 20 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us --- Chris Lohfink On Sep 17, 2014, at 6:09 PM, Yatong Zhang <bluefl...@gmail.com> wrote: > My sstable size is 192MB. I removed some data directories to reduce the data > that need to load, and this time it worked, so I was sure this was because of > the data was too large. > I tried to tune the JVM parameters, like heap size or stack size, but didn't > help. I finally got it resolved by add some options to '/etc/sysctl.conf': > > # Controls the maximum number of PID > kernel.pid_max = 9999999 > # Controls the maximum number of threads > kernel.threads-max = 9999999 > # Controls the maximum number of virtual memory areas > vm.max_map_count = 9999999 > > Hope this would be helpful to others, but any other advices are also welcome > > On Wed, Sep 17, 2014 at 11:54 PM, Rahul Neelakantan <ra...@rahul.be> wrote: > What is your sstable size set to for each of the sstables, using LCS? Are you > at the default of 5 MB? > > Rahul Neelakantan > > On Sep 17, 2014, at 10:58 AM, Yatong Zhang <bluefl...@gmail.com> wrote: > >> sorry, about 300k+ >> >> On Wed, Sep 17, 2014 at 10:56 PM, Yatong Zhang <bluefl...@gmail.com> wrote: >> no, I am running 64 bit JVM。 But I have many sstable files, about 30k+ >> >> On Wed, Sep 17, 2014 at 10:50 PM, graham sanderson <gra...@vast.com> wrote: >> Are you running on a 32 bit JVM? >> >> On Sep 17, 2014, at 9:43 AM, Yatong Zhang <bluefl...@gmail.com> wrote: >> >>> Hi there, >>> >>> I am using leveled compaction strategy and have many sstable files. The >>> error was during the startup, so any idea about this? >>> >>> ERROR [FlushWriter:4] 2014-09-17 22:36:59,383 CassandraDaemon.java (line >>> 199) Exception in thread Thread[FlushWriter:4,5,main] >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:693) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) >>> at >>> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:724) >>> ERROR [FlushWriter:2] 2014-09-17 22:36:59,472 CassandraDaemon.java (line >>> 199) Exception in thread Thread[FlushWriter:2,5,main] >>> FSReadError in >>> /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-23-Index.db >>> at >>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200) >>> at >>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168) >>> at >>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334) >>> at >>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:324) >>> at >>> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:394) >>> at >>> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342) >>> at >>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) >>> at >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:724) >>> Caused by: java.io.IOException: Map failed >>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849) >>> at >>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192) >>> ... 10 more >>> Caused by: java.lang.OutOfMemoryError: Map failed >>> at sun.nio.ch.FileChannelImpl.map0(Native Method) >>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846) >>> ... 11 more >>> >> >> >> >