Hi,

We use these type of crashes as indicator that the node might have some
hardware errors.

Did you check the ram? (eg memtest86)


Thibaut



On Wed, Nov 2, 2011 at 2:03 PM, Jahangir Mohammed
<[email protected]>wrote:

> Hello All,
>
> JVM is crashing on the cassandra nodes. Re-start doesn't help for long.
>
> Ring information:
> $ bin/nodetool -h A ring;
> Address         DC          Rack        Status State   Load
>  Owns    Token
>
>      127605887595351923798765477786913079297
> A   DC1         RAC1        Up     Normal  83.65 GB        25.00%  0
> B    DC2         RAC1        Down   Normal  170.09 GB       0.00%   1
> C   DC1         RAC1        Up     Normal  94.6 GB         25.00%
>  42535295865117307932921825928971026432
> D    DC2         RAC1        Up     Normal  87 GB           0.00%
> 42535295865117307932921825928971026433
> E   DC1         RAC1        Up     Normal  98.05 GB        25.00%
>  85070591730234615865843651857942052864
> F    DC2         RAC1        Up     Normal  95.55 GB        0.00%
> 85070591730234615865843651857942052865
> G   DC1         RAC1        Up     Normal  111.22 GB       25.00%
>  127605887595351923798765477786913079296
> H    DC2         RAC1        Up     Normal  42.05 GB        0.00%
> 127605887595351923798765477786913079297
>
> Details:
> 10GB Heap space.
> Memory on each node = 98 GB
> Disk space on each node = 400 GB
>
> JVM Crashes with segmentation faults. Have to do frequent re-starts of the
> nodes.
> Space on B is 170 GB and is getting CPU bound on re-start. but didn't get
> added to ring for almost 7 hours now.
>
> Java version:
>  java -version
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>
> JVM Crash Error log:
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00002abc7ec41fbc, pid=14232, tid=1104185664
> #
> # JRE version: 6.0_24-b07
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # V  [libjvm.so+0x30ffbc]
> #
> # If you would like to submit a bug report, please visit:
> #   http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
> ---------------  T H R E A D  ---------------
>
> Current thread (0x000000004d374000):  GCTaskThread [stack:
> 0x0000000000000000,0x0000000000000000] [id=14243]
>
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> si_addr=0x0000000000000010
>
> Registers:
>
>
> Any ideas/suggestions? Any preferred JVM version? There is nothing in
> cassandra logs to identify what's going on.
>
> Thanks,
> Jahangir.
>

Reply via email to