On Wed, Nov 2, 2011 at 2:03 PM, Jahangir Mohammed <md.jahangi...@gmail.com> wrote: > Hello All, > JVM is crashing on the cassandra nodes. Re-start doesn't help for long. > Ring information: > $ bin/nodetool -h A ring; > Address DC Rack Status State Load Owns > Token > > 127605887595351923798765477786913079297 > A DC1 RAC1 Up Normal 83.65 GB 25.00% 0 > B DC2 RAC1 Down Normal 170.09 GB 0.00% 1 > C DC1 RAC1 Up Normal 94.6 GB 25.00% > 42535295865117307932921825928971026432 > D DC2 RAC1 Up Normal 87 GB 0.00% > 42535295865117307932921825928971026433 > E DC1 RAC1 Up Normal 98.05 GB 25.00% > 85070591730234615865843651857942052864 > F DC2 RAC1 Up Normal 95.55 GB 0.00% > 85070591730234615865843651857942052865 > G DC1 RAC1 Up Normal 111.22 GB 25.00% > 127605887595351923798765477786913079296 > H DC2 RAC1 Up Normal 42.05 GB 0.00% > 127605887595351923798765477786913079297 > Details: > 10GB Heap space. > Memory on each node = 98 GB > Disk space on each node = 400 GB > JVM Crashes with segmentation faults. Have to do frequent re-starts of the > nodes. > Space on B is 170 GB and is getting CPU bound on re-start. but didn't get > added to ring for almost 7 hours now. > Java version: > java -version > java version "1.6.0_24" > Java(TM) SE Runtime Environment (build 1.6.0_24-b07) > Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) > JVM Crash Error log: > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x00002abc7ec41fbc, pid=14232, tid=1104185664 > # > # JRE version: 6.0_24-b07 > # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode > linux-amd64 compressed oops) > # Problematic frame: > # V [libjvm.so+0x30ffbc] > # > # If you would like to submit a bug report, please visit: > # http://java.sun.com/webapps/bugreport/crash.jsp > # > --------------- T H R E A D --------------- > Current thread (0x000000004d374000): GCTaskThread [stack: > 0x0000000000000000,0x0000000000000000] [id=14243] > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), > si_addr=0x0000000000000010 > Registers: > > Any ideas/suggestions? Any preferred JVM version? There is nothing in > cassandra logs to identify what's going on. And the update is pretty old,
That's a JVM segfault so has nothing to do with Cassandra. And hotspot update 19 is pretty old so that's not so surprising. You usually want to go with as recent as you can to avoid those but if memory servers there is known problem with JVM < 21 that Cassandra hits. But really I would go with update 27 or 29 without much hesitation. Btw, Cassandra 0.8.1 is pretty old too. > Thanks, > Jahangir.
