@Chris Lohfink I have 16G memory per node, all the other settings are default
@J. Ryan Earl I am not sure. I am using the default settings. But I've found out it might be because some settings in '/etc/sysctl.conf'. I am still testing it On Wed, Sep 17, 2014 at 11:46 AM, J. Ryan Earl <o...@jryanearl.us> wrote: > Are you using JNA? Did you adjust your memlock limit? > > On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <clohf...@blackbirdit.com> > wrote: > >> How much memory does your system have? How much memory is system >> utilizing before starting Cassandra (use command free)? What are the heap >> setting it tries to use? >> >> Chris >> >> On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bluefl...@gmail.com> wrote: >> >> It's during the startup. I tried to upgrade cassandra from 2.0.7 to >> 2.0.10, but looks like cassandra could not start again. Also I found the >> following log at '/var/log/messages': >> >> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more >>> than 120 seconds. >>> Sep 16 09:06:59 storage6 kernel: Tainted: G >>> --------------- H 2.6.32-431.el6.x86_64 #1 >>> Sep 16 09:06:59 storage6 kernel: "echo 0 > >>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0 >>> 4971 1 0x00000080 >>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082 >>> ffffffff81ed4ff0 ffff8803b4f01540 >>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370 >>> ffff88042b591ca0 ffff8803b4f01540 >>> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8 >>> 000000000000fbc8 ffff8803b4f01af8 >>> Sep 16 09:06:59 storage6 kernel: Call Trace: >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? >>> exit_robust_list+0x90/0x160 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] >>> do_group_exit+0x58/0xd0 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] >>> get_signal_to_deliver+0x1f6/0x460 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] >>> do_signal+0x75/0x800 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ? >>> wake_up_new_task+0xd9/0x130 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ? >>> do_fork+0x13d/0x480 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? >>> sys_futex+0x7b/0x170 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] >>> do_notify_resume+0x90/0xc0 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] >>> int_signal+0x12/0x17 >>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more >>> than 120 seconds. >>> Sep 16 09:06:59 storage6 kernel: Tainted: G >>> --------------- H 2.6.32-431.el6.x86_64 #1 >>> Sep 16 09:06:59 storage6 kernel: "echo 0 > >>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0 >>> 4972 1 0x00000080 >>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082 >>> ffffffff81ed6d78 ffff8803b4cf1500 >>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370 >>> ffff8803b4d7fca0 ffff8803b4cf1500 >>> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8 >>> 000000000000fbc8 ffff8803b4cf1ab8 >>> Sep 16 09:06:59 storage6 kernel: Call Trace: >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? >>> exit_robust_list+0x90/0x160 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ? >>> wake_up_state+0x10/0x20 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] >>> do_group_exit+0x58/0xd0 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] >>> get_signal_to_deliver+0x1f6/0x460 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] >>> do_signal+0x75/0x800 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ? >>> __switch_to+0x1ac/0x320 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ? >>> thread_return+0x4e/0x76e >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? >>> sys_futex+0x7b/0x170 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] >>> do_notify_resume+0x90/0xc0 >>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] >>> int_signal+0x12/0x17 >>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more >>> than 120 seconds. >>> >> >> >> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> >> wrote: >> >>> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bluefl...@gmail.com> >>> wrote: >>> >>>> I just encountered an error which left a log '/hs_err_pid3013.log'. So >>>> is there a way to solve this? >>>> >>>> # There is insufficient memory for the Java Runtime Environment to >>>>> continue. >>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for >>>>> committing reserved memory. >>>>> >>>> >>> Use less heap memory? >>> >>> You haven't specified under which circumstances this occurred, so I can >>> only conjecture that it is likely being caused by "writing too fast". >>> >>> Write more slowly. >>> >>> =Rob >>> >>> >> >> >
