You should upgrade to the latest Sun JVM. OpenJDK is almost a year behind in bug fixes.
On Fri, May 13, 2011 at 11:40 AM, James Cipar <jci...@cmu.edu> wrote: > It is a 64 bit VM. I didn't notice the hs_err_pid.log files since I'm > staring over ssh, so they're in my home directory instead of my working > directory. I've attached one of those below. I don't know much about Java, > so I'm not sure how to interpret this file. > > > > > > # > # An unexpected error has been detected by Java Runtime Environment: > # > # Internal Error (nmethod.cpp:1707), pid=10068, tid=1085823312 > # Error: guarantee(cont_offset != 0,"unhandled implicit exception in > compiled code") > # > # Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux-amd64) > # If you would like to submit a bug report, please visit: > # http://icedtea.classpath.org/bugzilla > # > > --------------- T H R E A D --------------- > > Current thread (0x00007f7e441a9c00): JavaThread "EXPIRING-MAP-TIMER-1" > daemon [_thread_in_Java, id=10095, > stack(0x0000000040b65000,0x0000000040b86000)] > > Stack: [0x0000000040b65000,0x0000000040b86000], sp=0x0000000040b83eb0, free > space=123k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > V [libjvm.so+0x5d198f] > V [libjvm.so+0x5d1cf1] > V [libjvm.so+0x27e506] > V [libjvm.so+0x4970d6] > V [libjvm.so+0x514cd9] > V [libjvm.so+0x4b0b64] > > > --------------- P R O C E S S --------------- > > Java Threads: ( => current thread ) > 0x0000000002913c00 JavaThread "NonPeriodicTasks:1" [_thread_blocked, > id=10233, stack(0x00000000401fc000,0x000000004021d000)] > 0x00000000026eec00 JavaThread "FILEUTILS-DELETE-POOL:1" [_thread_blocked, > id=10222, stack(0x0000000042533000,0x0000000042554000)] > 0x00007f7e3c01c000 JavaThread "Thread-22" [_thread_in_native, id=10212, > stack(0x0000000042512000,0x0000000042533000)] > 0x000000000253c000 JavaThread "Thread-21" [_thread_in_native, id=10211, > stack(0x0000000040dfc000,0x0000000040e1d000)] > 0x000000000253a800 JavaThread "Thread-20" [_thread_in_native, id=10210, > stack(0x0000000040ddb000,0x0000000040dfc000)] > 0x0000000002537400 JavaThread "Thread-19" [_thread_in_native, id=10209, > stack(0x00000000424f1000,0x0000000042512000)] > 0x0000000002725c00 JavaThread "pool-1-thread-1" [_thread_in_native, > id=10208, stack(0x000000004007d000,0x000000004009e000)] > 0x00000000026f7400 JavaThread "Thread-18" [_thread_in_native, id=10207, > stack(0x000000004034f000,0x0000000040370000)] > 0x0000000002902800 JavaThread "Thread-17" [_thread_in_native, id=10206, > stack(0x0000000040d6b000,0x0000000040d8c000)] > 0x0000000002901400 JavaThread "Thread-16" [_thread_in_native, id=10205, > stack(0x00000000424d0000,0x00000000424f1000)] > 0x0000000002613c00 JavaThread "Thread-15" [_thread_in_native, id=10204, > stack(0x000000004024e000,0x000000004026f000)] > 0x00000000026ad800 JavaThread "Thread-14" [_thread_in_native, id=10203, > stack(0x0000000040d11000,0x0000000040d32000)] > 0x000000000276f000 JavaThread "Thread-13" [_thread_in_native, id=10202, > stack(0x00000000424af000,0x00000000424d0000)] > 0x00000000026b2c00 JavaThread "Thread-12" [_thread_in_native, id=10201, > stack(0x0000000040490000,0x00000000404b1000)] > 0x00000000026aec00 JavaThread "Thread-11" [_thread_in_native, id=10200, > stack(0x000000004248e000,0x00000000424af000)] > 0x000000000254ec00 JavaThread "Thread-10" [_thread_in_native, id=10199, > stack(0x000000004246d000,0x000000004248e000)] > 0x000000000254d000 JavaThread "Thread-9" [_thread_in_native, id=10198, > stack(0x000000004244c000,0x000000004246d000)] > 0x0000000002505000 JavaThread "Thread-8" [_thread_in_native, id=10197, > stack(0x000000004242b000,0x000000004244c000)] > 0x0000000002502400 JavaThread "Thread-7" [_thread_in_native, id=10196, > stack(0x000000004240a000,0x000000004242b000)] > 0x0000000002500400 JavaThread "WRITE-/172.19.149.80" [_thread_blocked, > id=10195, stack(0x00000000407dd000,0x00000000407fe000)] > 0x00000000024ff000 JavaThread "WRITE-/172.19.149.80" [_thread_blocked, > id=10194, stack(0x0000000040c8e000,0x0000000040caf000)] > 0x00000000024f4400 JavaThread "WRITE-/172.19.149.64" [_thread_blocked, > id=10193, stack(0x00000000423e9000,0x000000004240a000)] > 0x00000000024f3000 JavaThread "WRITE-/172.19.149.64" [_thread_blocked, > id=10192, stack(0x00000000423c8000,0x00000000423e9000)] > 0x00000000024f1400 JavaThread "WRITE-/172.19.149.71" [_thread_blocked, > id=10191, stack(0x0000000040a2d000,0x0000000040a4e000)] > 0x00000000024f0000 JavaThread "WRITE-/172.19.149.71" [_thread_blocked, > id=10190, stack(0x00000000423a7000,0x00000000423c8000)] > 0x00000000024ee400 JavaThread "WRITE-/172.19.149.62" [_thread_blocked, > id=10189, stack(0x0000000042386000,0x00000000423a7000)] > 0x00000000026bf800 JavaThread "WRITE-/172.19.149.62" [_thread_blocked, > id=10188, stack(0x0000000042365000,0x0000000042386000)] > 0x00000000026bdc00 JavaThread "WRITE-/172.19.149.72" [_thread_blocked, > id=10187, stack(0x0000000040dba000,0x0000000040ddb000)] > 0x00000000026bc400 JavaThread "WRITE-/172.19.149.72" [_thread_blocked, > id=10186, stack(0x0000000042344000,0x0000000042365000)] > 0x00000000026bac00 JavaThread "WRITE-/172.19.149.63" [_thread_blocked, > id=10185, stack(0x0000000040431000,0x0000000040452000)] > 0x00000000026b9400 JavaThread "WRITE-/172.19.149.63" [_thread_blocked, > id=10184, stack(0x0000000042323000,0x0000000042344000)] > 0x00000000026b8000 JavaThread "WRITE-/172.19.149.61" [_thread_blocked, > id=10183, stack(0x0000000042302000,0x0000000042323000)] > 0x00000000024b9c00 JavaThread "WRITE-/172.19.149.61" [_thread_blocked, > id=10182, stack(0x0000000040c6d000,0x0000000040c8e000)] > 0x00000000024b8400 JavaThread "WRITE-/172.19.149.69" [_thread_blocked, > id=10181, stack(0x00000000422e1000,0x0000000042302000)] > 0x00000000024b7400 JavaThread "WRITE-/172.19.149.69" [_thread_blocked, > id=10180, stack(0x00000000422c0000,0x00000000422e1000)] > 0x00000000024b3800 JavaThread "Thread-6" [_thread_in_native, id=10179, > stack(0x0000000040e98000,0x0000000040eb9000)] > 0x00000000026e6400 JavaThread "Thread-5" [_thread_in_native, id=10178, > stack(0x0000000041222000,0x0000000041243000)] > 0x00007f7e4420ec00 JavaThread "ReadRepairStage:8" [_thread_blocked, > id=10177, stack(0x00000000407bc000,0x00000000407dd000)] > 0x00007f7e4420d400 JavaThread "ReadRepairStage:7" [_thread_blocked, > id=10176, stack(0x000000004229f000,0x00000000422c0000)] > 0x00007f7e4420c000 JavaThread "ReadRepairStage:6" [_thread_blocked, > id=10175, stack(0x000000004227e000,0x000000004229f000)] > 0x00007f7e4420a800 JavaThread "ReadRepairStage:5" [_thread_blocked, > id=10174, stack(0x00000000418ec000,0x000000004190d000)] > 0x00007f7e44209400 JavaThread "ReadRepairStage:4" [_thread_blocked, > id=10173, stack(0x0000000040a0c000,0x0000000040a2d000)] > 0x00007f7e44208000 JavaThread "ReadRepairStage:3" [_thread_blocked, > id=10172, stack(0x000000004225d000,0x000000004227e000)] > 0x00007f7e44206800 JavaThread "ReadRepairStage:2" [_thread_blocked, > id=10171, stack(0x000000004223c000,0x000000004225d000)] > 0x00007f7e44205400 JavaThread "ReadRepairStage:1" [_thread_blocked, > id=10170, stack(0x00000000418cb000,0x00000000418ec000)] > 0x00007f7e44203c00 JavaThread "MiscStage:1" [_thread_blocked, id=10169, > stack(0x00000000418aa000,0x00000000418cb000)] > 0x00007f7e44202800 JavaThread "MigrationStage:1" [_thread_blocked, id=10168, > stack(0x0000000041889000,0x00000000418aa000)] > 0x00007f7e44201000 JavaThread "AntiEntropyStage:1" [_thread_blocked, > id=10167, stack(0x0000000041868000,0x0000000041889000)] > 0x00007f7e441ffc00 JavaThread "GossipStage:1" [_thread_blocked, id=10166, > stack(0x0000000041847000,0x0000000041868000)] > 0x00007f7e441fe400 JavaThread "StreamStage:1" [_thread_blocked, id=10165, > stack(0x00000000417fe000,0x000000004181f000)] > 0x00007f7e441fd000 JavaThread "InternalResponseStage:8" [_thread_blocked, > id=10164, stack(0x00000000417dd000,0x00000000417fe000)] > 0x00007f7e441fb800 JavaThread "InternalResponseStage:7" [_thread_blocked, > id=10163, stack(0x0000000040c4c000,0x0000000040c6d000)] > 0x00007f7e441fa400 JavaThread "InternalResponseStage:6" [_thread_blocked, > id=10162, stack(0x000000004177a000,0x000000004179b000)] > 0x00007f7e441f8c00 JavaThread "InternalResponseStage:5" [_thread_blocked, > id=10161, stack(0x000000004194e000,0x000000004196f000)] > 0x00007f7e441f7800 JavaThread "InternalResponseStage:4" [_thread_blocked, > id=10160, stack(0x000000004027f000,0x00000000402a0000)] > 0x00007f7e441f6000 JavaThread "InternalResponseStage:3" [_thread_blocked, > id=10159, stack(0x0000000040811000,0x0000000040832000)] > 0x00007f7e441f4c00 JavaThread "InternalResponseStage:2" [_thread_blocked, > id=10158, stack(0x000000004221b000,0x000000004223c000)] > 0x00007f7e441f3400 JavaThread "InternalResponseStage:1" [_thread_blocked, > id=10157, stack(0x00000000409eb000,0x0000000040a0c000)] > 0x00007f7e441f2000 JavaThread "RequestResponseStage:8" [_thread_blocked, > id=10156, stack(0x0000000041759000,0x000000004177a000)] > 0x00007f7e441f0800 JavaThread "RequestResponseStage:7" [_thread_blocked, > id=10155, stack(0x0000000041e76000,0x0000000041e97000)] > 0x00007f7e44315000 JavaThread "RequestResponseStage:6" [_thread_blocked, > id=10154, stack(0x0000000040bf1000,0x0000000040c12000)] > 0x00007f7e44313800 JavaThread "RequestResponseStage:5" [_thread_blocked, > id=10153, stack(0x0000000041668000,0x0000000041689000)] > 0x00007f7e44312400 JavaThread "RequestResponseStage:4" [_thread_blocked, > id=10152, stack(0x0000000040d99000,0x0000000040dba000)] > 0x00007f7e44310c00 JavaThread "RequestResponseStage:3" [_thread_blocked, > id=10151, stack(0x00000000409ca000,0x00000000409eb000)] > 0x00007f7e4430f800 JavaThread "RequestResponseStage:2" [_thread_blocked, > id=10150, stack(0x000000004079b000,0x00000000407bc000)] > 0x00007f7e4430e000 JavaThread "RequestResponseStage:1" [_thread_blocked, > id=10149, stack(0x0000000041621000,0x0000000041642000)] > 0x00007f7e4430cc00 JavaThread "ReadStage:8" [_thread_blocked, id=10148, > stack(0x00000000421fa000,0x000000004221b000)] > 0x00007f7e4430b400 JavaThread "ReadStage:7" [_thread_blocked, id=10147, > stack(0x0000000040738000,0x0000000040759000)] > 0x00007f7e4430a000 JavaThread "ReadStage:6" [_thread_blocked, id=10146, > stack(0x0000000041264000,0x0000000041285000)] > 0x00007f7e44308800 JavaThread "ReadStage:5" [_thread_blocked, id=10145, > stack(0x0000000041e55000,0x0000000041e76000)] > 0x00007f7e44307400 JavaThread "ReadStage:4" [_thread_blocked, id=10144, > stack(0x0000000041243000,0x0000000041264000)] > 0x00007f7e44305c00 JavaThread "ReadStage:3" [_thread_blocked, id=10143, > stack(0x0000000041600000,0x0000000041621000)] > 0x00007f7e44304800 JavaThread "ReadStage:2" [_thread_blocked, id=10142, > stack(0x00000000415df000,0x0000000041600000)] > 0x00007f7e44303000 JavaThread "ReadStage:1" [_thread_blocked, id=10141, > stack(0x0000000041c20000,0x0000000041c41000)] > 0x00007f7e44301c00 JavaThread "MutationStage:32" [_thread_blocked, id=10140, > stack(0x0000000041738000,0x0000000041759000)] > 0x00007f7e44300400 JavaThread "MutationStage:31" [_thread_blocked, id=10139, > stack(0x0000000040982000,0x00000000409a3000)] > 0x00007f7e442ff000 JavaThread "MutationStage:30" [_thread_blocked, id=10138, > stack(0x00000000415be000,0x00000000415df000)] > 0x00007f7e442fdc00 JavaThread "MutationStage:29" [_thread_blocked, id=10137, > stack(0x000000004032e000,0x000000004034f000)] > 0x00007f7e442fc400 JavaThread "MutationStage:28" [_thread_blocked, id=10136, > stack(0x0000000041826000,0x0000000041847000)] > 0x00007f7e442fb000 JavaThread "MutationStage:27" [_thread_blocked, id=10135, > stack(0x0000000041bff000,0x0000000041c20000)] > 0x00007f7e442f9800 JavaThread "MutationStage:26" [_thread_blocked, id=10134, > stack(0x0000000040c2b000,0x0000000040c4c000)] > 0x00007f7e442f8400 JavaThread "MutationStage:25" [_thread_blocked, id=10133, > stack(0x0000000040117000,0x0000000040138000)] > 0x00007f7e442f6c00 JavaThread "MutationStage:24" [_thread_blocked, id=10132, > stack(0x0000000040ba7000,0x0000000040bc8000)] > 0x00007f7e442f5800 JavaThread "MutationStage:23" [_thread_blocked, id=10131, > stack(0x0000000040bd0000,0x0000000040bf1000)] > 0x00007f7e442f4400 JavaThread "MutationStage:22" [_thread_blocked, id=10130, > stack(0x0000000041717000,0x0000000041738000)] > 0x00007f7e442f2c00 JavaThread "MutationStage:21" [_thread_blocked, id=10129, > stack(0x00000000417bc000,0x00000000417dd000)] > 0x00007f7e442f1800 JavaThread "MutationStage:20" [_thread_blocked, id=10128, > stack(0x0000000040d4a000,0x0000000040d6b000)] > 0x00007f7e442f0000 JavaThread "MutationStage:19" [_thread_blocked, id=10127, > stack(0x0000000040564000,0x0000000040585000)] > 0x00007f7e442eec00 JavaThread "MutationStage:18" [_thread_blocked, id=10126, > stack(0x0000000040410000,0x0000000040431000)] > 0x00007f7e442ed400 JavaThread "MutationStage:17" [_thread_blocked, id=10125, > stack(0x0000000040717000,0x0000000040738000)] > 0x00007f7e442ec000 JavaThread "MutationStage:16" [_thread_blocked, id=10124, > stack(0x0000000040b86000,0x0000000040ba7000)] > 0x00007f7e442eac00 JavaThread "MutationStage:15" [_thread_blocked, id=10123, > stack(0x0000000041c75000,0x0000000041c96000)] > 0x00007f7e442e9400 JavaThread "MutationStage:14" [_thread_blocked, id=10122, > stack(0x00000000405ce000,0x00000000405ef000)] > 0x00007f7e442e8000 JavaThread "MutationStage:13" [_thread_blocked, id=10121, > stack(0x0000000041c41000,0x0000000041c62000)] > 0x00007f7e442e6800 JavaThread "MutationStage:12" [_thread_blocked, id=10120, > stack(0x0000000041e34000,0x0000000041e55000)] > 0x00007f7e442e5400 JavaThread "MutationStage:11" [_thread_blocked, id=10119, > stack(0x00000000421d9000,0x00000000421fa000)] > 0x00007f7e442e3c00 JavaThread "MutationStage:10" [_thread_blocked, id=10118, > stack(0x00000000416f6000,0x0000000041717000)] > 0x00007f7e4434cc00 JavaThread "MutationStage:9" [_thread_blocked, id=10117, > stack(0x00000000409a9000,0x00000000409ca000)] > 0x0000000002799000 JavaThread "MutationStage:8" [_thread_blocked, id=10116, > stack(0x0000000041bde000,0x0000000041bff000)] > 0x0000000002797800 JavaThread "MutationStage:7" [_thread_blocked, id=10115, > stack(0x00000000405ad000,0x00000000405ce000)] > 0x0000000002796000 JavaThread "MutationStage:6" [_thread_blocked, id=10114, > stack(0x0000000041e13000,0x0000000041e34000)] > 0x0000000002794800 JavaThread "MutationStage:5" [_thread_blocked, id=10113, > stack(0x0000000041df2000,0x0000000041e13000)] > 0x0000000002792c00 JavaThread "MutationStage:4" [_thread_blocked, id=10112, > stack(0x0000000040cf0000,0x0000000040d11000)] > 0x0000000002791c00 JavaThread "MutationStage:3" [_thread_blocked, id=10111, > stack(0x0000000041dd1000,0x0000000041df2000)] > 0x0000000002790c00 JavaThread "MutationStage:2" [_thread_blocked, id=10110, > stack(0x000000004077a000,0x000000004079b000)] > 0x0000000002790400 JavaThread "MutationStage:1" [_thread_blocked, id=10109, > stack(0x000000004159d000,0x00000000415be000)] > 0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10069, > stack(0x0000000040543000,0x0000000040564000)] > 0x00007f7e4434c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10069, > stack(0x0000000040543000,0x0000000040564000)] > 0x00007f7e44151c00 JavaThread "Thread-3" [_thread_in_native, id=10107, > stack(0x0000000040759000,0x000000004077a000)] > 0x00007f7e441e4000 JavaThread "HintedHandoff:1" [_thread_blocked, id=10106, > stack(0x0000000040178000,0x0000000040199000)] > 0x00000000026e7c00 JavaThread "WRITE-/172.19.149.83" [_thread_blocked, > id=10105, stack(0x00000000402f8000,0x0000000040319000)] > 0x00000000026e7000 JavaThread "WRITE-/172.19.149.83" [_thread_blocked, > id=10104, stack(0x000000004179b000,0x00000000417bc000)] > 0x0000000002700c00 JavaThread "ACCEPT-172-19-149-76/172.19.149.76" > [_thread_in_native, id=10103, stack(0x0000000041647000,0x0000000041668000)] > 0x00007f7e44162000 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=10102, > stack(0x0000000041201000,0x0000000041222000)] > 0x00007f7e441ccc00 JavaThread "CompactionExecutor:1" [_thread_blocked, > id=10101, stack(0x00000000416d5000,0x00000000416f6000)] > 0x0000000002676c00 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked, > id=10100, stack(0x000000004046f000,0x0000000040490000)] > 0x00000000026b7000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked, > id=10099, stack(0x00000000406f6000,0x0000000040717000)] > 0x00007f7e441bdc00 JavaThread "MemtablePostFlusher:1" [_thread_blocked, > id=10098, stack(0x000000004095a000,0x000000004097b000)] > 0x00007f7e4414c400 JavaThread "FlushWriter:1" [_thread_blocked, id=10097, > stack(0x0000000040ccf000,0x0000000040cf0000)] > 0x00007f7e4414b800 JavaThread "FlushSorter:1" [_thread_blocked, id=10096, > stack(0x0000000041aa6000,0x0000000041ac7000)] > =>0x00007f7e441a9c00 JavaThread "EXPIRING-MAP-TIMER-1" daemon > [_thread_in_Java, id=10095, stack(0x0000000040b65000,0x0000000040b86000)] > 0x00007f7e441c3400 JavaThread "ScheduledTasks:1" [_thread_blocked, id=10094, > stack(0x0000000040157000,0x0000000040178000)] > 0x00007f7e44152c00 JavaThread "FileWatchdog" daemon [_thread_blocked, > id=10093, stack(0x0000000040939000,0x000000004095a000)] > 0x00007f7e44146000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, > id=10091, stack(0x0000000041a85000,0x0000000041aa6000)] > 0x00007f7e44127c00 JavaThread "RMI TCP Accept-8080" daemon > [_thread_in_native, id=10090, stack(0x00000000401db000,0x00000000401fc000)] > 0x00007f7e44126800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, > id=10089, stack(0x00000000403ef000,0x0000000040410000)] > 0x00007f7e4402f000 JavaThread "Low Memory Detector" daemon [_thread_blocked, > id=10088, stack(0x0000000041699000,0x00000000416ba000)] > 0x00007f7e4402d000 JavaThread "CompilerThread1" daemon [_thread_blocked, > id=10087, stack(0x000000004149c000,0x000000004159d000)] > 0x00007f7e4402b800 JavaThread "CompilerThread0" daemon [_thread_blocked, > id=10086, stack(0x00000000420d8000,0x00000000421d9000)] > 0x00007f7e4402a000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, > id=10085, stack(0x00000000401ba000,0x00000000401db000)] > 0x00007f7e44028c00 JavaThread "Surrogate Locker Thread (CMS)" daemon > [_thread_blocked, id=10084, stack(0x000000004191b000,0x000000004193c000)] > 0x0000000002512c00 JavaThread "Finalizer" daemon [_thread_blocked, id=10083, > stack(0x0000000041da9000,0x0000000041dca000)] > 0x0000000002511400 JavaThread "Reference Handler" daemon [_thread_blocked, > id=10082, stack(0x00000000420b7000,0x00000000420d8000)] > > Other Threads: > 0x000000000250c000 VMThread [stack: 0x0000000040a64000,0x0000000040b65000] > [id=10081] > 0x00007f7e44147c00 WatcherThread [stack: > 0x00000000405f5000,0x00000000406f6000] [id=10092] > > VM state:not at safepoint (normal execution) > > VM Mutex/Monitor currently owned by a thread: None > > Heap > par new generation total 737280K, used 216364K [0x00007f7e548a0000, > 0x00007f7e868a0000, 0x00007f7e868a0000) > eden space 655360K, 20% used [0x00007f7e548a0000, 0x00007f7e5cbeb350, > 0x00007f7e7c8a0000) > from space 81920K, 100% used [0x00007f7e7c8a0000, 0x00007f7e818a0000, > 0x00007f7e818a0000) > to space 81920K, 0% used [0x00007f7e818a0000, 0x00007f7e818a0000, > 0x00007f7e868a0000) > concurrent mark-sweep generation total 6718464K, used 1264492K > [0x00007f7e868a0000, 0x00007f80209a0000, 0x00007f8020aa0000) > concurrent-mark-sweep perm gen total 30432K, used 18222K > [0x00007f8020aa0000, 0x00007f8022858000, 0x00007f8025ea0000) > > Dynamic libraries: > 00400000-00408000 r-xp 00000000 03:01 8576 > /usr/lib/jvm/java-6-openjdk/jre/bin/java > 00608000-00609000 rw-p 00008000 03:01 8576 > /usr/lib/jvm/java-6-openjdk/jre/bin/java > 023ba000-0416e000 rw-p 023ba000 00:00 0 > [heap] > ... > ... > [ Not sure how important this stuff is. There was a lot of it, if it's > important I can send it. > ... > 7f802b6f9000-7f802b6fc000 rw-p 7f802b6f9000 00:00 0 > 7f802b6fc000-7f802b6fe000 rw-p 0001b000 03:01 82264 > /lib/ld-2.7.so > 7fff7210c000-7fff72120000 rwxp 7ffffffe9000 00:00 0 > [stack] > 7fff72121000-7fff72121000 rw-p 7fffffffe000 00:00 0 > 7fff721ff000-7fff72200000 r-xp 7fff721ff000 00:00 0 > [vdso] > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > [vsyscall] > > VM Arguments: > jvm_args: -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms7361M > -Xmx7361M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true > -Dcom.sun.management.jmxremote.port=8080 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true > java_command: org.apache.cassandra.thrift.CassandraDaemon > Launcher Type: SUN_STANDARD > > Environment Variables: > PATH=/h/jcipar/SOFTWARE/ROOTS/Linux/x86_64/bin:/h/jcipar/bin:/h/jcipar/SOFTWARE/ROOTS/All/bin:/h/jcipar/SOFTWARE/ant/apache-ant-1.8.1/bin/:~mabdelm/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64 > SHELL=/bin/bash > > Signal Handlers: > SIGSEGV: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGBUS: [libjvm.so+0x5d2630], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGFPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGPIPE: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGXFSZ: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGILL: [libjvm.so+0x4ab9d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 > SIGUSR2: [libjvm.so+0x4ab380], sa_mask[0]=0x00000000, sa_flags=0x10000004 > SIGHUP: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 > SIGTERM: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGQUIT: [libjvm.so+0x4ad520], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > > > --------------- S Y S T E M --------------- > > OS:5.0.6 > > uname:Linux 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64 > libc:glibc 2.7 NPTL 2.7 > rlimit: STACK 8192k, CORE 0k, NPROC 124096, NOFILE 1024, AS infinity > load average:3.09 3.56 3.72 > > CPU:total 8 (1 cores per cpu, 1 threads per core) family 6 model 2 stepping > 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3 > > Memory: 4k page, physical 15075756k(6082384k free), swap 0k(0k free) > > vm_info: OpenJDK 64-Bit Server VM (1.6.0_0-b11) for linux-amd64 JRE > (1.6.0_0-b11), built on Apr 9 2009 19:35:18 by "pbuilder" with gcc 4.3.2 > > time: Tue May 10 13:01:39 2011 > elapsed time: 2175 seconds > > > > > > > > > > > On May 12, 2011, at 9:30 PM, Jeffrey Kesselman wrote: > >> If this a 64bit VM? >> >> A 32bit Java VM with default c-heap settings can only actually use >> about 2GB of Java Heap. >> >> On Thu, May 12, 2011 at 8:08 PM, James Cipar <jci...@cmu.edu> wrote: >>> Oh, forgot this detail: I have no swap configured, so swapping is not the >>> cause of the crash. Could it be that I'm running out of memory on a 15GB >>> machine? That seems unlikely. I grepped dmesg for "oom" and didn't see >>> anything from the oom killer, and I used the instructions from the >>> following web page and didn't see that the oom killer had killed anything. >>> >>> http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer >>> >>> jcipar@172-19-149-62:~$ sudo cat /var/log/messages | grep --ignore-case >>> "killed process" >>> jcipar@172-19-149-62:~$ >>> >>> >>> >>> Also, this is pretty subjective, so I can't say for sure until it finishes, >>> but this seems to be running *much* slower after setting the heap size and >>> setting up JNA. >>> >>> >>> >>> On May 12, 2011, at 7:52 PM, James Cipar wrote: >>> >>>> It looks like MAX_HEAP_SIZE is set in cassandra-env.sh to be half of my >>>> physical memory. These are 15GB VMs, so that's 7.5GB for Cassandra. I >>>> would have expected that to work, but I will override to 13 GB just to see >>>> what happens. >>>> >>>> I've also got the JNA thing set up. Do you think this would cause the >>>> crashes, or is it just a performance improvement? >>>> >>>> >>>> >>>> On May 12, 2011, at 7:27 PM, Sameer Farooqui wrote: >>>> >>>>> The key JVM options for Cassandra are in cassandra.in.sh. >>>>> >>>>> What is your min and max heap size? >>>>> >>>>> The default setting of max heap size is 1GB. How much RAM do your nodes >>>>> have? You may want to increase this setting. You can also set the -Xmx >>>>> and -Xms options to the same value to keep Java from having to manage >>>>> heap growth. On a 32-bit machine, you can get a max of about 1.6 GB of >>>>> heap; you can get a lot more on 64-bit. >>>>> >>>>> Try messing with some of the other settings in the cassandra.in.sh file. >>>>> >>>>> You may not have DEBUG mode turned on for Cassandra and therefore may not >>>>> be getting the full details of what's going on when the server crashes. >>>>> In the <cassandra-home>/conf/log4j-server.properties file, set this line >>>>> from the default of INFO to DEBUG: >>>>> >>>>> log4j.rootLogger=INFO,stdout,R >>>>> >>>>> >>>>> Also, you haven't configured JNA on this server. Here's some info about >>>>> it and how to configure it: >>>>> >>>>> JNA provides Java programs easy access to native shared libraries without >>>>> writing anything but Java code. >>>>> >>>>> Note from Cassandra developers for why JNA is needed: >>>>> "Linux aggressively swaps out infrequently used memory to make more room >>>>> for its file system buffer cache. Unfortunately, modern generational >>>>> garbage collectors like the JVM's leave parts of its heap un-touched for >>>>> relatively large amounts of time, leading Linux to swap it out. When the >>>>> JVM finally goes to use or GC that memory, swap hell ensues. >>>>> >>>>> Setting swappiness to zero can mitigate this behavior but does not >>>>> eliminate it entirely. Turning off swap entirely is effective. But to >>>>> avoid surprising people who don't know about this behavior, the best >>>>> solution is to tell Linux not to swap out the JVM, and that is what we do >>>>> now with mlockall via JNA. >>>>> >>>>> Because of licensing issues, we can't distribute JNA with Cassandra, so >>>>> you must manually add it to the Cassandra lib/ directory or otherwise >>>>> place it on the classpath. If the JNA jar is not present, Cassandra will >>>>> continue as before." >>>>> >>>>> Get JNA with: >>>>> cd ~ >>>>> wget >>>>> http://debian.riptano.com/debian/pool/libjna-java_3.2.7-0~nmu.2_amd64.deb >>>>> >>>>> To install: >>>>> techlabs@cassandraN1:~$ sudo dpkg -i libjna-java_3.2.7-0~nmu.2_amd64.deb >>>>> (Reading database ... 44334 files and directories currently installed.) >>>>> Preparing to replace libjna-java 3.2.4-2 (using >>>>> libjna-java_3.2.7-0~nmu.2_amd64.deb) ... >>>>> Unpacking replacement libjna-java ... >>>>> Setting up libjna-java (3.2.7-0~nmu.2) ... >>>>> >>>>> >>>>> The deb package will install the JNA jar file to /usr/share/java/jna.jar, >>>>> but Cassandra only loads it if its in the class path. The easy way to do >>>>> this is just create a symlink into your Cassandra lib directory (note: >>>>> replace /home/techlabs with your home dir location): >>>>> ln -s /usr/share/java/jna.jar /home/techlabs/apache-cassandra-0.7.0/lib >>>>> >>>>> Research: >>>>> http://journal.paul.querna.org/articles/2010/11/11/enabling-jna-in-cassandra/ >>>>> >>>>> >>>>> - Sameer >>>>> >>>>> >>>>> On Thu, May 12, 2011 at 4:15 PM, James Cipar <jci...@cmu.edu> wrote: >>>>> I'm using Cassandra 0.7.5, and uploading about 200 GB of data total (20 >>>>> GB unique data), to a cluster of 10 servers. I'm using batch_mutate, and >>>>> breaking the data up into chunks of about 10k records. Each record is >>>>> about 5KB, so a total of about 50MB per batch. When I upload a smaller 2 >>>>> GB data set, everything works fine. When I upload the 20 GB data set, >>>>> servers will occasionally crash. Currently I have my client code >>>>> automatically detect this and restart the server, but that is less than >>>>> ideal. >>>>> >>>>> I'm not sure what information to gather to determine what's going on >>>>> here. Here is a sample of a log file from when a crash occurred. The >>>>> crash was immediately after the log entry tagged "2011-05-12 >>>>> 19:02:19,377". Any idea what's going on here? Any other info I can >>>>> gather to try to debug this? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:07,855 GCInspector.java (line >>>>> 128) GC for ParNew: 375 ms, 576641232 reclaimed leaving 5471432144 used; >>>>> max is 7774142464 >>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:08,857 GCInspector.java (line >>>>> 128) GC for ParNew: 450 ms, -63738232 reclaimed leaving 5546942544 used; >>>>> max is 7774142464 >>>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:10,652 CommitLogSegment.java >>>>> (line 50) Creating new commitlog segment >>>>> /mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241330652.log >>>>> INFO [MutationStage:24] 2011-05-12 19:02:10,680 ColumnFamilyStore.java >>>>> (line 1070) Enqueuing flush of Memtable-Standard1@1256245282(51921529 >>>>> bytes, 1115783 operations) >>>>> INFO [FlushWriter:1] 2011-05-12 19:02:10,680 Memtable.java (line 158) >>>>> Writing Memtable-Standard1@1256245282(51921529 bytes, 1115783 operations) >>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:12,932 GCInspector.java (line >>>>> 128) GC for ParNew: 249 ms, 571827736 reclaimed leaving 3165899760 used; >>>>> max is 7774142464 >>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:15,253 GCInspector.java (line >>>>> 128) GC for ParNew: 341 ms, 561823592 reclaimed leaving 1764208800 used; >>>>> max is 7774142464 >>>>> INFO [FlushWriter:1] 2011-05-12 19:02:16,743 Memtable.java (line 165) >>>>> Completed flushing >>>>> /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-74-Data.db >>>>> (53646223 bytes) >>>>> INFO [COMMIT-LOG-WRITER] 2011-05-12 19:02:16,745 CommitLog.java (line >>>>> 440) Discarding obsolete commit >>>>> log:CommitLogSegment(/mnt/scratch/jcipar/cassandra/commitlog/CommitLog-1305241306438.log) >>>>> INFO [ScheduledTasks:1] 2011-05-12 19:02:18,256 GCInspector.java (line >>>>> 128) GC for ParNew: 305 ms, 544491840 reclaimed leaving 865198712 used; >>>>> max is 7774142464 >>>>> INFO [MutationStage:19] 2011-05-12 19:02:19,000 ColumnFamilyStore.java >>>>> (line 1070) Enqueuing flush of Memtable-Standard1@479849353(51941121 >>>>> bytes, 1115783 operations) >>>>> INFO [FlushWriter:1] 2011-05-12 19:02:19,000 Memtable.java (line 158) >>>>> Writing Memtable-Standard1@479849353(51941121 bytes, 1115783 operations) >>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,310 SSTable.java (line 147) >>>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-51 >>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,324 SSTable.java (line 147) >>>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-55 >>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,339 SSTable.java (line 147) >>>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-58 >>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,357 SSTable.java (line 147) >>>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-67 >>>>> INFO [NonPeriodicTasks:1] 2011-05-12 19:02:19,377 SSTable.java (line 147) >>>>> Deleted /mnt/scratch/jcipar/cassandra/data/Keyspace1/Standard1-f-61 >>>>> INFO [main] 2011-05-12 19:02:21,026 AbstractCassandraDaemon.java (line >>>>> 78) Logging initialized >>>>> INFO [main] 2011-05-12 19:02:21,040 AbstractCassandraDaemon.java (line >>>>> 96) Heap size: 7634681856/7635730432 >>>>> INFO [main] 2011-05-12 19:02:21,042 CLibrary.java (line 61) JNA not >>>>> found. Native methods will be disabled. >>>>> INFO [main] 2011-05-12 19:02:21,052 DatabaseDescriptor.java (line 121) >>>>> Loading settings from >>>>> file:/h/jcipar/Projects/HP/OtherDBs/Cassandra/apache-cassandra-0.7.5/conf/cassandra.yaml >>>>> INFO [main] 2011-05-12 19:02:21,178 DatabaseDescriptor.java (line 181) >>>>> DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap >>>>> INFO [main] 2011-05-12 19:02:21,310 SSTableReader.java (line 154) Opening >>>>> /mnt/scratch/jcipar/cassandra/data/system/Schema-f-1 >>>>> INFO [main] 2011-05-12 19:02:21,327 SSTableReader.java (line 154) Opening >>>>> /mnt/scratch/jcipar/cassandra/data/system/Schema-f-2 >>>>> INFO [main] 2011-05-12 19:02:21,336 SSTableReader.java (line 154) Opening >>>>> /mnt/scratch/jcipar/cassandra/data/system/Migrations-f-1 >>>>> INFO [main] 2011-05-12 19:02:21,337 SSTableReader.java (line 154) Opening >>>>> /mnt/scratch/jcipar/cassandra/data/system/Migrations-f-2 >>>>> INFO [main] 2011-05-12 19:02:21,342 SSTableReader.java (line 154) Opening >>>>> /mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-2 >>>>> INFO [main] 2011-05-12 19:02:21,344 SSTableReader.java (line 154) Opening >>>>> /mnt/scratch/jcipar/cassandra/data/system/LocationInfo-f-1 >>>>> INFO [main] 2011-05-12 19:02:21,379 DatabaseDescriptor.java (line 461) >>>>> Loading schema version 9467ffe0-7cea-11e0-8ddc-f74ef74e382f >>>>> >>>> >>>> >>> >>> >> >> >> >> -- >> It's always darkest just before you are eaten by a grue. >> > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
