It is a 64 bit VM. I didn't notice the hs_err_pid.log files since I'm staring over ssh, so they're in my home directory instead of my working directory. I've attached one of those below. I don't know much about Java, so I'm not sure how to interpret this file.
# # An unexpected error has been detected by Java Runtime Environment: # # Internal Error (nmethod.cpp:1707), pid=10068, tid=1085823312 # Error: guarantee(cont_offset != 0,"unhandled implicit exception in compiled code") # # Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux-amd64) # If you would like to submit a bug report, please visit: # http://icedtea.classpath.org/bugzilla # --------------- T H R E A D --------------- Current thread (0x00007f7e441a9c00): JavaThread "EXPIRING-MAP-TIMER-1" daemon [_thread_in_Java, id=10095, stack(0x0000000040b65000,0x0000000040b86000)] Stack: [0x0000000040b65000,0x0000000040b86000], sp=0x0000000040b83eb0, free space=123k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x5d198f] V [libjvm.so+0x5d1cf1] V [libjvm.so+0x27e506] V [libjvm.so+0x4970d6] V [libjvm.so+0x514cd9] V [libjvm.so+0x4b0b64] --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x0000000002913c00 JavaThread "NonPeriodicTasks:1" [_thread_blocked, id=10233, stack(0x00000000401fc000,0x000000004021d000)] 0x00000000026eec00 JavaThread "FILEUTILS-DELETE-POOL:1" [_thread_blocked, id=10222, stack(0x0000000042533000,0x0000000042554000)] 0x00007f7e3c01c000 JavaThread "Thread-22" [_thread_in_native, id=10212, stack(0x0000000042512000,0x0000000042533000)] 0x000000000253c000 JavaThread "Thread-21" [_thread_in_native, id=10211, stack(0x0000000040dfc000,0x0000000040e1d000)] 0x000000000253a800 JavaThread "Thread-20" [_thread_in_native, id=10210, stack(0x0000000040ddb000,0x0000000040dfc000)] 0x0000000002537400 JavaThread "Thread-19" [_thread_in_native, id=10209, stack(0x00000000424f1000,0x0000000042512000)] 0x0000000002725c00 JavaThread "pool-1-thread-1" [_thread_in_native, id=10208, stack(0x000000004007d000,0x000000004009e000)] 0x00000000026f7400 JavaThread "Thread-18" [_thread_in_native, id=10207, stack(0x000000004034f000,0x0000000040370000)] 0x0000000002902800 JavaThread "Thread-17" [_thread_in_native, id=10206, stack(0x0000000040d6b000,0x0000000040d8c000)] 0x0000000002901400 JavaThread "Thread-16" [_thread_in_native, id=10205, stack(0x00000000424d0000,0x00000000424f1000)] 0x0000000002613c00 JavaThread "Thread-15" [_thread_in_native, id=10204, stack(0x000000004024e000,0x000000004026f000)] 0x00000000026ad800 JavaThread "Thread-14" [_thread_in_native, id=10203, stack(0x0000000040d11000,0x0000000040d32000)] 0x000000000276f000 JavaThread "Thread-13" [_thread_in_native, id=10202, stack(0x00000000424af000,0x00000000424d0000)] 0x00000000026b2c00 JavaThread "Thread-12" [_thread_in_native, id=10201, stack(0x0000000040490000,0x00000000404b1000)] 0x00000000026aec00 JavaThread "Thread-11" [_thread_in_native, id=10200, stack(0x000000004248e000,0x00000000424af000)] 0x000000000254ec00 JavaThread "Thread-10" [_thread_in_native, id=10199, stack(0x000000004246d000,0x000000004248e000)] 0x000000000254d000 JavaThread "Thread-9" [_thread_in_native, id=10198, stack(0x000000004244c000,0x000000004246d000)] 0x0000000002505000 JavaThread "Thread-8" [_thread_in_native, id=10197, stack(0x000000004242b000,0x000000004244c000)] 0x0000000002502400 JavaThread "Thread-7" [_thread_in_native, id=10196, stack(0x000000004240a000,0x000000004242b000)] 0x0000000002500400 JavaThread "WRITE-/172.19.149.80" [_thread_blocked, id=10195, stack(0x00000000407dd000,0x00000000407fe000)] 0x00000000024ff000 JavaThread "WRITE-/172.19.149.80" [_thread_blocked, id=10194, stack(0x0000000040c8e000,0x0000000040caf000)] 0x00000000024f4400 JavaThread "WRITE-/172.19.149.64" [_thread_blocked, id=10193, stack(0x00000000423e9000,0x000000004240a000)] 0x00000000024f3000 JavaThread "WRITE-/172.19.149.64" [_thread_blocked, id=10192, stack(0x00000000423c8000,0x00000000423e9000)] 0x00000000024f1400 JavaThread "WRITE-/172.19.149.71" [_thread_blocked, id=10191, stack(0x0000000040a2d000,0x0000000040a4e000)] 0x00000000024f0000 JavaThread "WRITE-/172.19.149.71" [_thread_blocked, id=10190, stack(0x00000000423a7000,0x00000000423c8000)] 0x00000000024ee400 JavaThread "WRITE-/172.19.149.62" [_thread_blocked, id=10189, stack(0x0000000042386000,0x00000000423a7000)] 0x00000000026bf800 JavaThread "WRITE-/172.19.149.62" [_thread_blocked, id=10188, stack(0x0000000042365000,0x0000000042386000)] 0x00000000026bdc00 JavaThread "WRITE-/172.19.149.72" [_thread_blocked, id=10187, stack(0x0000000040dba000,0x0000000040ddb000)] 0x00000000026bc400 JavaThread "WRITE-/172.19.149.72" [_thread_blocked, id=10186, stack(0x0000000042344000,0x0000000042365000)] 0x00000000026bac00 JavaThread "WRITE-/172.19.149.63" [_thread_blocked, id=10185, stack(0x0000000040431000,0x0000000040452000)] 0x00000000026b9400 JavaThread "WRITE-/172.19.149.63" [_thread_blocked, id=10184, stack(0x0000000042323000,0x0000000042344000)] 0x00000000026b8000 JavaThread "WRITE-/172.19.149.61" [_thread_blocked, id=10183, stack(0x0000000042302000,0x0000000042323000)] 0x00000000024b9c00 JavaThread "WRITE-/172.19.149.61" [_thread_blocked, id=10182, stack(0x0000000040c6d000,0x0000000040c8e000)] 0x00000000024b8400 JavaThread "WRITE-/172.19.149.69" [_thread_blocked, id=10181, stack(0x00000000422e1000,0x0000000042302000)] 0x00000000024b7400 JavaThread "WRITE-/172.19.149.69" [_thread_blocked, id=10180, stack(0x00000000422c0000,0x00000000422e1000)] 0x00000000024b3800 JavaThread "Thread-6" [_thread_in_native, id=10179, stack(0x0000000040e98000,0x0000000040eb9000)] 0x00000000026e6400 JavaThread "Thread-5" [_thread_in_native, id=10178, stack(0x0000000041222000,0x0000000041243000)] 0x00007f7e4420ec00 JavaThread "ReadRepairStage:8" [_thread_blocked, id=10177, stack(0x00000000407bc000,0x00000000407dd000)] 0x00007f7e4420d400 JavaThread "ReadRepairStage:7" [_thread_blocked, id=10176, stack(0x000000004229f000,0x00000000422c0000)] 0x00007f7e4420c000 JavaThread "ReadRepairStage:6" [_thread_blocked, id=10175, stack(0x000000004227e000,0x000000004229f000)] 0x00007f7e4420a800 JavaThread "ReadRepairStage:5" [_thread_blocked, id=10174, stack(0x00000000418ec000,0x000000004190d000)] 0x00007f7e44209400 JavaThread "ReadRepairStage:4" [_thread_blocked, id=10173, stack(0x0000000040a0c000,0x0000000040a2d000)] 0x00007f7e44208000 JavaThread "ReadRepairStage:3" [_thread_blocked, id=10172, stack(0x000000004225d000,0x000000004227e000)] 0x00007f7e44206800 JavaThread "ReadRepairStage:2" [_thread_blocked, id=10171, stack(0x000000004223c000,0x000000004225d000)] 0x00007f7e44205400 JavaThread "ReadRepairStage:1" [_thread_blocked, id=10170, stack(0x00000000418cb000,0x00000000418ec000)] 0x00007f7e44203c00 JavaThread "MiscStage:1" [_thread_blocked, id=10169, stack(0x00000000418aa000,0x00000000418cb000)] 0x00007f7e44202800 JavaThread "MigrationStage:1" [_thread_blocked, id=10168, stack(0x0000000041889000,0x00000000418aa000)] 0x00007f7e44201000 JavaThread "AntiEntropyStage:1" [_thread_blocked, id=10167, stack(0x0000000041868000,0x0000000041889000)] 0x00007f7e441ffc00 JavaThread "GossipStage:1" [_thread_blocked, id=10166, stack(0x0000000041847000,0x0000000041868000)] 0x00007f7e441fe400 JavaThread "StreamStage:1" [_thread_blocked, id=10165, stack(0x00000000417fe000,0x000000004181f000)] 0x00007f7e441fd000 JavaThread "InternalResponseStage:8" [_thread_blocked, id=10164, stack(0x00000000417dd000,0x00000000417fe000)] 0x00007f7e441fb800 JavaThread "InternalResponseStage:7" [_thread_blocked, id=10163, stack(0x0000000040c4c000,0x0000000040c6d000)] 0x00007f7e441fa400 JavaThread "InternalResponseStage:6" [_thread_blocked, id=10162, stack(0x000000004177a000,0x000000004179b000)] 0x00007f7e441f8c00 JavaThread "InternalResponseStage:5" [_thread_blocked, id=10161, stack(0x000000004194e000,0x000000004196f000)] 0x00007f7e441f7800 JavaThread "InternalResponseStage:4" [_thread_blocked, id=10160, stack(0x000000004027f000,0x00000000402a0000)] 0x00007f7e441f6000 JavaThread "InternalResponseStage:3" [_thread_blocked, id=10159, stack(0x0000000040811000,0x0000000040832000)] 0x00007f7e441f4c00 JavaThread "InternalResponseStage:2" [_thread_blocked, id=10158, stack(0x000000004221b000,0x000000004223c000)] 0x00007f7e441f3400 JavaThread "InternalResponseStage:1" [_thread_blocked, id=10157, stack(0x00000000409eb000,0x0000000040a0c000)] 0x00007f7e441f2000 JavaThread "RequestResponseStage:8" [_thread_blocked, id=10156, stack(0x0000000041759000,0x000000004177a000)] 0x00007f7e441f0800 JavaThread "RequestResponseStage:7" [_thread_blocked, id=10155, stack(0x0000000041e76000,0x0000000041e97000)] 0x00007f7e44315000 JavaThread "RequestResponseStage:6" [_thread_blocked, id=10154, stack(0x0000000040bf1000,0x0000000040c12000)] 0x00007f7e44313800 JavaThread "RequestResponseStage:5" [_thread_blocked, id=10153, stack(0x0000000041668000,0x0000000041689000)] 0x00007f7e44312400 JavaThread "RequestResponseStage:4" [_thread_blocked, id=10152, stack(0x0000000040d99000,0x0000000040dba000)] 0x00007f7e44310c00 JavaThread "RequestResponseStage:3" [_thread_blocked, id=10151, stack(0x00000000409ca000,0x00000000409eb000)] 0x00007f7e4430f800 JavaThread "RequestResponseStage:2" [_thread_blocked, id=10150, stack(0x000000004079b000,0x00000000407bc000)] 0x00007f7e4430e000 JavaThread "RequestResponseStage:1" [_thread_blocked, id=10149, stack(0x0000000041621000,0x0000000041642000)] 0x00007f7e4430cc00 JavaThread "ReadStage:8" [_thread_blocked, id=10148, stack(0x00000000421fa000,0x000000004221b000)] 0x00007f7e4430b400 JavaThread "ReadStage:7" [_thread_blocked, id=10147, stack(0x0000000040738000,0x0000000040759000)] 0x00007f7e4430a000 JavaThread "ReadStage:6" [_thread_blocked, id=10146, stack(0x0000000041264000,0x0000000041285000)] 0x00007f7e44308800 JavaThread "ReadStage:5" [_thread_blocked, id=10145, stack(0x0000000041e55000,0x0000000041e76000)] 0x00007f7e44307400 JavaThread "ReadStage:4" [_thread_blocked, id=10144, stack(0x0000000041243000,0x0000000041264000)] 0x00007f7e44305c00 JavaThread "ReadStage:3" [_thread_blocked, id=10143, stack(0x0000000041600000,0x0000000041621000)] 0x00007f7e44304800 JavaThread "ReadStage:2" [_thread_blocked, id=10142, stack(0x00000000415df000,0x0000000041600000)] 0x00007f7e44303000 JavaThread "ReadStage:1" [_thread_blocked, id=10141, stack(0x0000000041c20000,0x0000000041c41000)] 0x00007f7e44301c00 JavaThread "MutationStage:32" [_thread_blocked, id=10140, stack(0x0000000041738000,0x0000000041759000)] 0x00007f7e44300400 JavaThread "MutationStage:31" [_thread_blocked, id=10139, stack(0x0000000040982000,0x00000000409a3000)] 0x00007f7e442ff000 JavaThread "MutationStage:30" [_thread_blocked, id=10138, stack(0x00000000415be000,0x00000000415df000)] 0x00007f7e442fdc00 JavaThread "MutationStage:29" [_thread_blocked, id=10137, stack(0x000000004032e000,0x000000004034f000)] 0x00007f7e442fc400 JavaThread "MutationStage:28" [_thread_blocked, id=10136, stack(0x0000000041826000,0x0000000041847000)] 0x00007f7e442fb000 JavaThread "MutationStage:27" [_thread_blocked, id=10135, stack(0x0000000041bff000,0x0000000041c20000)] 0x00007f7e442f9800 JavaThread "MutationStage:26" [_thread_blocked, id=10134, stack(0x0000000040c2b000,0x0000000040c4c000)] 0x00007f7e442f8400 JavaThread "MutationStage:25" [_thread_blocked, id=10133, stack(0x0000000040117000,0x0000000040138000)] 0x00007f7e442f6c00 JavaThread "MutationStage:24" [_thread_blocked, id=10132, stack(0x0000000040ba7000,0x0000000040bc8000)] 0x00007f7e442f5800 JavaThread "MutationStage:23" [_thread_blocked, id=10131, stack(0x0000000040bd0000,0x0000000040bf1000)] 0x00007f7e442f4400 JavaThread "MutationStage:22" [_thread_blocked, id=10130, stack(0x0000000041717000,0x0000000041738000)] 0x00007f7e442f2c00 JavaThread "MutationStage:21" [_thread_blocked, id=10129, stack(0x00000000417bc000,0x00000000417dd000)] 0x00007f7e442f1800 JavaThread "MutationStage:20" [_thread_blocked, id=10128, stack(0x0000000040d4a000,0x0000000040d6b000)] 0x00007f7e442f0000 JavaThread "MutationStage:19" [_thread_blocked, id=10127, stack(0x0000000040564000,0x0000000040585000)] 0x00007f7e442eec00 JavaThread "MutationStage:18" [_thread_blocked, id=10126, stack(0x0000000040410000,0x0000000040431000)] 0x00007f7e442ed400 JavaThread "MutationStage:17" [_thread_blocked, id=10125, stack(0x0000000040717000,0x0000000040738000)] 0x00007f7e442ec000 JavaThread "MutationStage:16" [_thread_blocked, id=10124, stack(0x0000000040b86000,0x0000000040ba7000)] 0x00007f7e442eac00 JavaThread "MutationStage:15" [_thread_blocked, id=10123, stack(0x0000000041c75000,0x0000000041c96000)] 0x00007f7e442e9400 JavaThread "MutationStage:14" [_thread_blocked, id=10122, stack(0x00000000405ce000,0x00000000405ef000)] 0x00007f7e442e8000 JavaThread "MutationStage:13" [_thread_blocked, id=10121, stack(0x0000000041c41000,0x0000000041c62000)] 0x00007f7e442e6800 JavaThread "MutationStage:12" [_thread_blocked, id=10120, stack(0x0000000041e34000,0x0000000041e55000)] 0x00007f7e442e5400 JavaThread "MutationStage:11" [_thread_blocked, id=10119, stack(0x00000000421d9000,0x00000000421fa000)] 0x00007f7e442e3c00 JavaThread "MutationStage:10" [_thread_blocked, id=10118, stack(0x00000000416f6000,0x0000000041717000)] 0x00007f7e4434cc00 JavaThread "MutationStage:9" [_thread_blocked, id=10117, stack(0x00000000409a9000,0x00000000409ca000)] 0x0000000002799000 JavaThread "MutationStage:8" [_thread_blocked, id=10116, stack(0x0000000041bde000,0x0000000041bff000)] 0x0000000002797800 JavaThread "MutationStage:7" [_thread_blocked, id=10115, stack(0x00000000405ad000,0x00000000405ce000)] 0x0000000002796000 JavaThread "MutationStage:6" [_thread_blocked, id=10114, stack(0x0000000041e13000,0x0000000041e34000)] 0x0000000002794800 JavaThread "MutationStage:5" [_thread_blocked, id=10113, stack(0x0000000041df2000,0x0000000041e13000)] 0x0000000002792c00 JavaThread "MutationStage:4" [_thread_blocked, id=10112, stack(0x0000000040cf0000,0x0000000040d11000)] 0x0000000002791c00 JavaThread "MutationStage:3" [_thread_blocked, id=10111, stack(0x0000000041dd1000,0x0000000041df2000)] 0x0000000002790c00 JavaThread "MutationStage:2" [_thread_blocked, id=10110, stack(0x000000004077a000,0x000000004079b000)] 0x0000000002790400 JavaThread "MutationStage:1" [_thread_blocked, id=10109, stack(0x000000004159d000,0x00000000415be000)] 0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10069, stack(0x0000000040543000,0x0000000040564000)] 0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10069, stack(0x0000000040543000,0x0000000040564000)] 0x00007f7e44151c00 JavaThread "Thread-3" [_thread_in_native, id=10107, stack(0x0000000040759000,0x000000004077a000)] 0x00007f7e441e4000 JavaThread "HintedHandoff:1" [_thread_blocked, id=10106, stack(0x0000000040178000,0x0000000040199000)] 0x00000000026e7c00 JavaThread "WRITE-/172.19.149.83" [_thread_blocked, id=10105, stack(0x00000000402f8000,0x0000000040319000)] 0x00000000026e7000 JavaThread "WRITE-/172.19.149.83" [_thread_blocked, id=10104, stack(0x000000004179b000,0x00000000417bc000)] 0x0000000002700c00 JavaThread "ACCEPT-172-19-149-76/172.19.149.76" [_thread_in_native, id=10103, stack(0x0000000041647000,0x0000000041668000)] 0x00007f7e44162000 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=10102, stack(0x0000000041201000,0x0000000041222000)] 0x00007f7e441ccc00 JavaThread "CompactionExecutor:1" [_thread_blocked, id=10101, stack(0x00000000416d5000,0x00000000416f6000)] 0x0000000002676c00 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked, id=10100, stack(0x000000004046f000,0x0000000040490000)] 0x00000000026b7000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked, id=10099, stack(0x00000000406f6000,0x0000000040717000)] 0x00007f7e441bdc00 JavaThread "MemtablePostFlusher:1" [_thread_blocked, id=10098, stack(0x000000004095a000,0x000000004097b000)] 0x00007f7e4414c400 JavaThread "FlushWriter:1" [_thread_blocked, id=10097, stack(0x0000000040ccf000,0x0000000040cf0000)] 0x00007f7e4414b800 JavaThread "FlushSorter:1" [_thread_blocked, id=10096, stack(0x0000000041aa6000,0x0000000041ac7000)] =>0x00007f7e441a9c00 JavaThread "EXPIRING-MAP-TIMER-1" daemon [_thread_in_Java, id=10095, stack(0x0000000040b65000,0x0000000040b86000)] 0x00007f7e441c3400 JavaThread "ScheduledTasks:1" [_thread_blocked, id=10094, stack(0x0000000040157000,0x0000000040178000)] 0x00007f7e44152c00 JavaThread "FileWatchdog" daemon [_thread_blocked, id=10093, stack(0x0000000040939000,0x000000004095a000)] 0x00007f7e44146000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=10091, stack(0x0000000041a85000,0x0000000041aa6000)] 0x00007f7e44127c00 JavaThread "RMI TCP Accept-8080" daemon [_thread_in_native, id=10090, stack(0x00000000401db000,0x00000000401fc000)] 0x00007f7e44126800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=10089, stack(0x00000000403ef000,0x0000000040410000)] 0x00007f7e4402f000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=10088, stack(0x0000000041699000,0x00000000416ba000)] 0x00007f7e4402d000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=10087, stack(0x000000004149c000,0x000000004159d000)] 0x00007f7e4402b800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=10086, stack(0x00000000420d8000,0x00000000421d9000)] 0x00007f7e4402a000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10085, stack(0x00000000401ba000,0x00000000401db000)] 0x00007f7e44028c00 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=10084, stack(0x000000004191b000,0x000000004193c000)] 0x0000000002512c00 JavaThread "Finalizer" daemon [_thread_blocked, id=10083, stack(0x0000000041da9000,0x0000000041dca000)] 0x0000000002511400 JavaThread "Reference Handler" daemon [_thread_blocked, id=10082, stack(0x00000000420b7000,0x00000000420d8000)] Other Threads: 0x000000000250c000 VMThread [stack: 0x0000000040a64000,0x0000000040b65000] [id=10081] 0x00007f7e44147c00 WatcherThread [stack: 0x00000000405f5000,0x00000000406f6000] [id=10092] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap par new generation total 737280K, used 216364K [0x00007f7e548a0000, 0x00007f7e868a0000, 0x00007f7e868a0000) eden space 655360K, 20% used [0x00007f7e548a0000, 0x00007f7e5cbeb350, 0x00007f7e7c8a0000) from space 81920K, 100% used [0x00007f7e7c8a0000, 0x00007f7e818a0000, 0x00007f7e818a0000) to space 81920K, 0% used [0x00007f7e818a0000, 0x00007f7e818a0000, 0x00007f7e868a0000) concurrent mark-sweep generation total 6718464K, used 1264492K [0x00007f7e868a0000, 0x00007f80209a0000, 0x00007f8020aa0000) concurrent-mark-sweep perm gen total 30432K, used 18222K [0x00007f8020aa0000, 0x00007f8022858000, 0x00007f8025ea0000) Dynamic libraries: 00400000-00408000 r-xp 00000000 03:01 8576 /usr/lib/jvm/java-6-openjdk/jre/bin/java 00608000-00609000 rw-p 00008000 03:01 8576 /usr/lib/jvm/java-6-openjdk/jre/bin/java 023ba000-0416e000 rw-p 023ba000 00:00 0 [heap] ... ... [ Not sure how important this stuff is. There was a lot of it, if it's important I can send it. ... 7f802b6f9000-7f802b6fc000 rw-p 7f802b6f9000 00:00 0 7f802b6fc000-7f802b6fe000 rw-p 0001b000 03:01 82264 /lib/ld-2.7.so 7fff7210c000-7fff72120000 rwxp 7ffffffe9000 00:00 0 [stack] 7fff72121000-7fff72121000 rw-p 7fffffffe000 00:00 0 7fff721ff000-7fff72200000 r-xp 7fff721ff000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] VM Arguments: jvm_args: -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms7361M -Xmx7361M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=8080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true java_command: org.apache.cassandra.thrift.CassandraDaemon Launcher Type: SUN_STANDARD Environment Variables: PATH=/h/jcipar/SOFTWARE/ROOTS/Linux/x86_64/bin:/h/jcipar/bin:/h/jcipar/SOFTWARE/ROOTS/All/bin:/h/jcipar/SOFTWARE/ant/apache-ant-1.8.1/bin/:~mabdelm/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64 SHELL=/bin/bash Signal Handlers: SIGSEGV: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGBUS: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGFPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGPIPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGXFSZ: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGILL: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x4ab380], sa_mask[0]=0x00000000, sa_flags=0x10000004 SIGHUP: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGTERM: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGQUIT: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 --------------- S Y S T E M --------------- OS:5.0.6 uname:Linux 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64 libc:glibc 2.7 NPTL 2.7 rlimit: STACK 8192k, CORE 0k, NPROC 124096, NOFILE 1024, AS infinity load average:3.09 3.56 3.72 CPU:total 8 (1 cores per cpu, 1 threads per core) family 6 model 2 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3 Memory: 4k page, physical 15075756k(6082384k free), swap 0k(0k free) vm_info: OpenJDK 64-Bit Server VM (1.6.0_0-b11) for linux-amd64 JRE (1.6.0_0-b11), built on Apr 9 2009 19:35:18 by "pbuilder" with gcc 4.3.2 time: Tue May 10 13:01:39 2011 elapsed time: 2175 seconds On May 12, 2011, at 9:30 PM, Jeffrey Kesselman wrote: > If this a 64bit VM? > > A 32bit Java VM with default c-heap settings can only actually use > about 2GB of Java Heap. > > On Thu, May 12, 2011 at 8:08 PM, James Cipar <jci...@cmu.edu> wrote: >> Oh, forgot this detail: I have no swap configured, so swapping is not the >> cause of the crash. Could it be that I'm running out of memory on a 15GB >> machine? That seems unlikely. I grepped dmesg for "oom" and didn't see >> anything from the oom killer, and I used the instructions from the following >> web page and didn't see that the oom killer had killed anything. >> >> http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer >> >> jcipar@172-19-149-62:~$ sudo cat /var/log/messages | grep --ignore-case >> "killed process" >> jcipar@172-19-149-62:~$ >> >> >> >> Also, this is pretty subjective, so I can't say for sure until it finishes, >> but this seems to be running *much* slower after setting the heap size and >> setting up JNA. >> >> >> >> On May 12, 2011, at 7:52 PM, James Cipar wrote: >> >>> It looks like MAX_HEAP_SIZE is set in cassandra-env.sh to be half of my >>> physical memory. These are 15GB VMs, so that's 7.5GB for Cassandra. I >>> would have expected that to work, but I will override to 13 GB just to see >>> what happens. >>> >>> I've also got the JNA thing set up. Do you think this would cause the >>> crashes, or is it just a performance improvement? >>> >>> >>> >>> On May 12, 2011, at 7:27 PM, Sameer Farooqui wrote: >>> >>>> The key JVM options for Cassandra are in cassandra.in.sh. >>>> >>>> What is your min and max heap size? >>>> >>>> The default setting of max heap size is 1GB. How much RAM do your nodes >>>> have? You may want to increase this setting. You can also set the -Xmx and >>>> -Xms options to the same value to keep Java from having to manage heap >>>> growth. On a 32-bit machine, you can get a max of about 1.6 GB of heap; >>>> you can get a lot more on 64-bit. >>>> >>>> Try messing with some of the other settings in the cassandra.in.sh file. >>>> >>>> You may not have DEBUG mode turned on for Cassandra and therefore may not >>>> be getting the full details of what's going on when the server crashes. In >>>> the <cassandra-home>/conf/log4j-server.properties file, set this line from >>>> the default of INFO to DEBUG: >>>> >>>> log4j.rootLogger=INFO,stdout,R >>>> >>>> >>>> Also, you haven't configured JNA on this server. Here's some info about it >>>> and how to configure it: >>>> >>>> JNA provides Java programs easy access to native shared libraries without >>>> writing anything but Java code. >>>> >>>> Note from Cassandra developers for why JNA is needed: >>>> "Linux aggressively swaps out infrequently used memory to make more room >>>> for its file system buffer cache. Unfortunately, modern generational >>>> garbage collectors like the JVM's leave parts of its heap un-touched for >>>> relatively large amounts of time, leading Linux to swap it out. When the >>>> JVM finally goes to use or GC that memory, swap hell ensues. >>>> >>>> Setting swappiness to zero can mitigate this behavior but does not >>>> eliminate it entirely. Turning off swap entirely is effective. But to >>>> avoid surprising people who don't know about this behavior, the best >>>> solution is to tell Linux not to swap out the JVM, and that is what we do >>>> now with mlockall via JNA. >>>> >>>> Because of licensing issues, we can't distribute JNA with Cassandra, so >>>> you must manually add it to the Cassandra lib/ directory or otherwise >>>> place it on the classpath. If the JNA jar is not present, Cassandra will >>>> continue as before." >>>> >>>> Get JNA with: >>>> cd ~ >>>> wget >>>> http://debian.riptano.com/debian/pool/libjna-java_3.2.7-0~nmu.2_amd64.deb >>>> >>>> To install: >>>> techlabs@cassandraN1:~$ sudo dpkg -i libjna-java_3.2.7-0~nmu.2_amd64.deb >>>> (Reading database ... 44334 files and directories currently installed.) >>>> Preparing to replace libjna-java 3.2.4-2 (using >>>> libjna-java_3.2.7-0~nmu.2_amd64.deb) ... >>>> Unpacking replacement libjna-java ... >>>> Setting up libjna-java (3.2.7-0~nmu.2) ... >>>> >>>> >>>> The deb package will install the JNA jar file to /usr/share/java/jna.jar, >>>> but Cassandra only loads it if its in the class path. The easy way to do >>>> this is just create a symlink into your Cassandra lib directory (note: >>>> replace /home/techlabs with your home dir location): >>>> ln -s /usr/share/java/jna.jar /home/techlabs/apache-cassandra-0.7.0/lib >>>> >>>> Research: >>>> http://journal.paul.querna.org/articles/2010/11/11/enabling-jna-in-cassandra/ >>>> >>>> >>>> - Sameer >>>> >>>> >>>> On Thu, May 12, 2011 at 4:15 PM, James Cipar <jci...@cmu.edu> wrote: >>>> I'm using Cassandra 0.7.5, and uploading about 200 GB of data total (20 GB >>>> unique data), to a cluster of 10 servers. I'm using batch_mutate, and >>>> breaking the data up into chunks of about 10k records. Each record is >>>> about 5KB, so a total of about 50MB per batch. When I upload a smaller 2 >>>> GB data set, everything works fine. When I upload the 20 GB data set, >>>> servers will occasionally crash. Currently I have my client code >>>> automatically detect this and restart the server, but that is less than >>>> ideal. >>>> >>>> I'm not sure what information to gather to determine what's going on here. >>>> Here is a sample of a log file from when a crash occurred. The crash was >>>> immediately after the log entry tagged "2011-05-12 19:02:19,377". Any >>>> idea what's going on here? Any other info I can gather to try to debug >>>> this? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:07,855 GCInspector.java (line >>>> 128) GC for ParNew: 375 ms, 576641232 reclaimed leaving 5471432144 used; >>>> max is 7774142464 >>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:08,857 GCInspector.java (line >>>> 128) GC for ParNew: 450 ms, -63738232 reclaimed leaving 5546942544 used; >>>> max is 7774142464 >>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:10,652 CommitLogSegment.java >>>> (line 50) Creating new commitlog segment >>>> /mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241330652.log >>>> INFO [MutationStage:24] 2011-05-12 19:02:10,680 ColumnFamilyStore.java >>>> (line 1070) Enqueuing flush of Memtable-Standard1@1256245282(51921529 >>>> bytes, 1115783 operations) >>>> INFO [FlushWriter:1] 2011-05-12 19:02:10,680 Memtable.java (line 158) >>>> Writing Memtable-Standard1@1256245282(51921529 bytes, 1115783 operations) >>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:12,932 GCInspector.java (line >>>> 128) GC for ParNew: 249 ms, 571827736 reclaimed leaving 3165899760 used; >>>> max is 7774142464 >>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:15,253 GCInspector.java (line >>>> 128) GC for ParNew: 341 ms, 561823592 reclaimed leaving 1764208800 used; >>>> max is 7774142464 >>>> INFO [FlushWriter:1] 2011-05-12 19:02:16,743 Memtable.java (line 165) >>>> Completed flushing >>>> /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-74-Data.db >>>> (53646223 bytes) >>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:16,745 CommitLog.java (line 440) >>>> Discarding obsolete commit >>>> log:CommitLogSegment(/mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241306438.log) >>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:18,256 GCInspector.java (line >>>> 128) GC for ParNew: 305 ms, 544491840 reclaimed leaving 865198712 used; >>>> max is 7774142464 >>>> INFO [MutationStage:19] 2011-05-12 19:02:19,000 ColumnFamilyStore.java >>>> (line 1070) Enqueuing flush of Memtable-Standard1@479849353(51941121 >>>> bytes, 1115783 operations) >>>> INFO [FlushWriter:1] 2011-05-12 19:02:19,000 Memtable.java (line 158) >>>> Writing Memtable-Standard1@479849353(51941121 bytes, 1115783 operations) >>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,310 SSTable.java (line 147) >>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-51 >>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,324 SSTable.java (line 147) >>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-55 >>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,339 SSTable.java (line 147) >>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-58 >>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,357 SSTable.java (line 147) >>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-67 >>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,377 SSTable.java (line 147) >>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-61 >>>> INFO [main] 2011-05-12 19:02:21,026 AbstractCassandraDaemon.java (line 78) >>>> Logging initialized >>>> INFO [main] 2011-05-12 19:02:21,040 AbstractCassandraDaemon.java (line 96) >>>> Heap size: 7634681856/7635730432 >>>> INFO [main] 2011-05-12 19:02:21,042 CLibrary.java (line 61) JNA not found. >>>> Native methods will be disabled. >>>> INFO [main] 2011-05-12 19:02:21,052 DatabaseDescriptor.java (line 121) >>>> Loading settings from >>>> file:/h/jcipar/Projects/HP/OtherDBs/Cassandra/apache-cassandra-0.7.5/conf/cassandra.yaml >>>> INFO [main] 2011-05-12 19:02:21,178 DatabaseDescriptor.java (line 181) >>>> DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap >>>> INFO [main] 2011-05-12 19:02:21,310 SSTableReader.java (line 154) Opening >>>> /mnt/scratch/jcipar/cassandra/data/system/Schema-f-1 >>>> INFO [main] 2011-05-12 19:02:21,327 SSTableReader.java (line 154) Opening >>>> /mnt/scratch/jcipar/cassandra/data/system/Schema-f-2 >>>> INFO [main] 2011-05-12 19:02:21,336 SSTableReader.java (line 154) Opening >>>> /mnt/scratch/jcipar/cassandra/data/system/Migrations-f-1 >>>> INFO [main] 2011-05-12 19:02:21,337 SSTableReader.java (line 154) Opening >>>> /mnt/scratch/jcipar/cassandra/data/system/Migrations-f-2 >>>> INFO [main] 2011-05-12 19:02:21,342 SSTableReader.java (line 154) Opening >>>> /mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-2 >>>> INFO [main] 2011-05-12 19:02:21,344 SSTableReader.java (line 154) Opening >>>> /mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-1 >>>> INFO [main] 2011-05-12 19:02:21,379 DatabaseDescriptor.java (line 461) >>>> Loading schema version 9467ffe0-7cea-11e0-8ddc-f74ef74e382f >>>> >>> >>> >> >> > > > > -- > It's always darkest just before you are eaten by a grue. >
