If this a 64bit VM? A 32bit Java VM with default c-heap settings can only actually use about 2GB of Java Heap.
On Thu, May 12, 2011 at 8:08 PM, James Cipar <jci...@cmu.edu> wrote: > Oh, forgot this detail: I have no swap configured, so swapping is not the > cause of the crash. Could it be that I'm running out of memory on a 15GB > machine? That seems unlikely. I grepped dmesg for "oom" and didn't see > anything from the oom killer, and I used the instructions from the following > web page and didn't see that the oom killer had killed anything. > > http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer > > jcipar@172-19-149-62:~$ sudo cat /var/log/messages | grep --ignore-case > "killed process" > jcipar@172-19-149-62:~$ > > > > Also, this is pretty subjective, so I can't say for sure until it finishes, > but this seems to be running *much* slower after setting the heap size and > setting up JNA. > > > > On May 12, 2011, at 7:52 PM, James Cipar wrote: > >> It looks like MAX_HEAP_SIZE is set in cassandra-env.sh to be half of my >> physical memory. These are 15GB VMs, so that's 7.5GB for Cassandra. I >> would have expected that to work, but I will override to 13 GB just to see >> what happens. >> >> I've also got the JNA thing set up. Do you think this would cause the >> crashes, or is it just a performance improvement? >> >> >> >> On May 12, 2011, at 7:27 PM, Sameer Farooqui wrote: >> >>> The key JVM options for Cassandra are in cassandra.in.sh. >>> >>> What is your min and max heap size? >>> >>> The default setting of max heap size is 1GB. How much RAM do your nodes >>> have? You may want to increase this setting. You can also set the -Xmx and >>> -Xms options to the same value to keep Java from having to manage heap >>> growth. On a 32-bit machine, you can get a max of about 1.6 GB of heap; you >>> can get a lot more on 64-bit. >>> >>> Try messing with some of the other settings in the cassandra.in.sh file. >>> >>> You may not have DEBUG mode turned on for Cassandra and therefore may not >>> be getting the full details of what's going on when the server crashes. In >>> the <cassandra-home>/conf/log4j-server.properties file, set this line from >>> the default of INFO to DEBUG: >>> >>> log4j.rootLogger=INFO,stdout,R >>> >>> >>> Also, you haven't configured JNA on this server. Here's some info about it >>> and how to configure it: >>> >>> JNA provides Java programs easy access to native shared libraries without >>> writing anything but Java code. >>> >>> Note from Cassandra developers for why JNA is needed: >>> "Linux aggressively swaps out infrequently used memory to make more room >>> for its file system buffer cache. Unfortunately, modern generational >>> garbage collectors like the JVM's leave parts of its heap un-touched for >>> relatively large amounts of time, leading Linux to swap it out. When the >>> JVM finally goes to use or GC that memory, swap hell ensues. >>> >>> Setting swappiness to zero can mitigate this behavior but does not >>> eliminate it entirely. Turning off swap entirely is effective. But to avoid >>> surprising people who don't know about this behavior, the best solution is >>> to tell Linux not to swap out the JVM, and that is what we do now with >>> mlockall via JNA. >>> >>> Because of licensing issues, we can't distribute JNA with Cassandra, so you >>> must manually add it to the Cassandra lib/ directory or otherwise place it >>> on the classpath. If the JNA jar is not present, Cassandra will continue as >>> before." >>> >>> Get JNA with: >>> cd ~ >>> wget >>> http://debian.riptano.com/debian/pool/libjna-java_3.2.7-0~nmu.2_amd64.deb >>> >>> To install: >>> techlabs@cassandraN1:~$ sudo dpkg -i libjna-java_3.2.7-0~nmu.2_amd64.deb >>> (Reading database ... 44334 files and directories currently installed.) >>> Preparing to replace libjna-java 3.2.4-2 (using >>> libjna-java_3.2.7-0~nmu.2_amd64.deb) ... >>> Unpacking replacement libjna-java ... >>> Setting up libjna-java (3.2.7-0~nmu.2) ... >>> >>> >>> The deb package will install the JNA jar file to /usr/share/java/jna.jar, >>> but Cassandra only loads it if its in the class path. The easy way to do >>> this is just create a symlink into your Cassandra lib directory (note: >>> replace /home/techlabs with your home dir location): >>> ln -s /usr/share/java/jna.jar /home/techlabs/apache-cassandra-0.7.0/lib >>> >>> Research: >>> http://journal.paul.querna.org/articles/2010/11/11/enabling-jna-in-cassandra/ >>> >>> >>> - Sameer >>> >>> >>> On Thu, May 12, 2011 at 4:15 PM, James Cipar <jci...@cmu.edu> wrote: >>> I'm using Cassandra 0.7.5, and uploading about 200 GB of data total (20 GB >>> unique data), to a cluster of 10 servers. I'm using batch_mutate, and >>> breaking the data up into chunks of about 10k records. Each record is >>> about 5KB, so a total of about 50MB per batch. When I upload a smaller 2 >>> GB data set, everything works fine. When I upload the 20 GB data set, >>> servers will occasionally crash. Currently I have my client code >>> automatically detect this and restart the server, but that is less than >>> ideal. >>> >>> I'm not sure what information to gather to determine what's going on here. >>> Here is a sample of a log file from when a crash occurred. The crash was >>> immediately after the log entry tagged "2011-05-12 19:02:19,377". Any idea >>> what's going on here? Any other info I can gather to try to debug this? >>> >>> >>> >>> >>> >>> >>> >>> INFO [ScheduledTasks:1] 2011-05-12 19:02:07,855 GCInspector.java (line 128) >>> GC for ParNew: 375 ms, 576641232 reclaimed leaving 5471432144 used; max is >>> 7774142464 >>> INFO [ScheduledTasks:1] 2011-05-12 19:02:08,857 GCInspector.java (line 128) >>> GC for ParNew: 450 ms, -63738232 reclaimed leaving 5546942544 used; max is >>> 7774142464 >>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:10,652 CommitLogSegment.java >>> (line 50) Creating new commitlog segment >>> /mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241330652.log >>> INFO [MutationStage:24] 2011-05-12 19:02:10,680 ColumnFamilyStore.java >>> (line 1070) Enqueuing flush of Memtable-Standard1@1256245282(51921529 >>> bytes, 1115783 operations) >>> INFO [FlushWriter:1] 2011-05-12 19:02:10,680 Memtable.java (line 158) >>> Writing Memtable-Standard1@1256245282(51921529 bytes, 1115783 operations) >>> INFO [ScheduledTasks:1] 2011-05-12 19:02:12,932 GCInspector.java (line 128) >>> GC for ParNew: 249 ms, 571827736 reclaimed leaving 3165899760 used; max is >>> 7774142464 >>> INFO [ScheduledTasks:1] 2011-05-12 19:02:15,253 GCInspector.java (line 128) >>> GC for ParNew: 341 ms, 561823592 reclaimed leaving 1764208800 used; max is >>> 7774142464 >>> INFO [FlushWriter:1] 2011-05-12 19:02:16,743 Memtable.java (line 165) >>> Completed flushing >>> /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-74-Data.db >>> (53646223 bytes) >>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:16,745 CommitLog.java (line 440) >>> Discarding obsolete commit >>> log:CommitLogSegment(/mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241306438.log) >>> INFO [ScheduledTasks:1] 2011-05-12 19:02:18,256 GCInspector.java (line 128) >>> GC for ParNew: 305 ms, 544491840 reclaimed leaving 865198712 used; max is >>> 7774142464 >>> INFO [MutationStage:19] 2011-05-12 19:02:19,000 ColumnFamilyStore.java >>> (line 1070) Enqueuing flush of Memtable-Standard1@479849353(51941121 bytes, >>> 1115783 operations) >>> INFO [FlushWriter:1] 2011-05-12 19:02:19,000 Memtable.java (line 158) >>> Writing Memtable-Standard1@479849353(51941121 bytes, 1115783 operations) >>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,310 SSTable.java (line 147) >>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-51 >>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,324 SSTable.java (line 147) >>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-55 >>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,339 SSTable.java (line 147) >>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-58 >>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,357 SSTable.java (line 147) >>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-67 >>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,377 SSTable.java (line 147) >>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-61 >>> INFO [main] 2011-05-12 19:02:21,026 AbstractCassandraDaemon.java (line 78) >>> Logging initialized >>> INFO [main] 2011-05-12 19:02:21,040 AbstractCassandraDaemon.java (line 96) >>> Heap size: 7634681856/7635730432 >>> INFO [main] 2011-05-12 19:02:21,042 CLibrary.java (line 61) JNA not found. >>> Native methods will be disabled. >>> INFO [main] 2011-05-12 19:02:21,052 DatabaseDescriptor.java (line 121) >>> Loading settings from >>> file:/h/jcipar/Projects/HP/OtherDBs/Cassandra/apache-cassandra-0.7.5/conf/cassandra.yaml >>> INFO [main] 2011-05-12 19:02:21,178 DatabaseDescriptor.java (line 181) >>> DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap >>> INFO [main] 2011-05-12 19:02:21,310 SSTableReader.java (line 154) Opening >>> /mnt/scratch/jcipar/cassandra/data/system/Schema-f-1 >>> INFO [main] 2011-05-12 19:02:21,327 SSTableReader.java (line 154) Opening >>> /mnt/scratch/jcipar/cassandra/data/system/Schema-f-2 >>> INFO [main] 2011-05-12 19:02:21,336 SSTableReader.java (line 154) Opening >>> /mnt/scratch/jcipar/cassandra/data/system/Migrations-f-1 >>> INFO [main] 2011-05-12 19:02:21,337 SSTableReader.java (line 154) Opening >>> /mnt/scratch/jcipar/cassandra/data/system/Migrations-f-2 >>> INFO [main] 2011-05-12 19:02:21,342 SSTableReader.java (line 154) Opening >>> /mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-2 >>> INFO [main] 2011-05-12 19:02:21,344 SSTableReader.java (line 154) Opening >>> /mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-1 >>> INFO [main] 2011-05-12 19:02:21,379 DatabaseDescriptor.java (line 461) >>> Loading schema version 9467ffe0-7cea-11e0-8ddc-f74ef74e382f >>> >> >> > > -- It's always darkest just before you are eaten by a grue.
