Are you using JNA? Did you adjust your memlock limit? On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <clohf...@blackbirdit.com> wrote:
> How much memory does your system have? How much memory is system utilizing > before starting Cassandra (use command free)? What are the heap setting it > tries to use? > > Chris > > On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bluefl...@gmail.com> wrote: > > It's during the startup. I tried to upgrade cassandra from 2.0.7 to > 2.0.10, but looks like cassandra could not start again. Also I found the > following log at '/var/log/messages': > > Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more >> than 120 seconds. >> Sep 16 09:06:59 storage6 kernel: Tainted: G >> --------------- H 2.6.32-431.el6.x86_64 #1 >> Sep 16 09:06:59 storage6 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0 >> 4971 1 0x00000080 >> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082 >> ffffffff81ed4ff0 ffff8803b4f01540 >> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370 >> ffff88042b591ca0 ffff8803b4f01540 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8 >> 000000000000fbc8 ffff8803b4f01af8 >> Sep 16 09:06:59 storage6 kernel: Call Trace: >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? >> exit_robust_list+0x90/0x160 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] >> do_group_exit+0x58/0xd0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] >> get_signal_to_deliver+0x1f6/0x460 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ? >> wake_up_new_task+0xd9/0x130 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ? >> do_fork+0x13d/0x480 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? >> sys_futex+0x7b/0x170 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] >> do_notify_resume+0x90/0xc0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17 >> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more >> than 120 seconds. >> Sep 16 09:06:59 storage6 kernel: Tainted: G >> --------------- H 2.6.32-431.el6.x86_64 #1 >> Sep 16 09:06:59 storage6 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0 >> 4972 1 0x00000080 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082 >> ffffffff81ed6d78 ffff8803b4cf1500 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370 >> ffff8803b4d7fca0 ffff8803b4cf1500 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8 >> 000000000000fbc8 ffff8803b4cf1ab8 >> Sep 16 09:06:59 storage6 kernel: Call Trace: >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? >> exit_robust_list+0x90/0x160 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ? >> wake_up_state+0x10/0x20 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] >> do_group_exit+0x58/0xd0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] >> get_signal_to_deliver+0x1f6/0x460 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ? >> __switch_to+0x1ac/0x320 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ? >> thread_return+0x4e/0x76e >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? >> sys_futex+0x7b/0x170 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] >> do_notify_resume+0x90/0xc0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17 >> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more >> than 120 seconds. >> > > > On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> wrote: > >> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bluefl...@gmail.com> >> wrote: >> >>> I just encountered an error which left a log '/hs_err_pid3013.log'. So >>> is there a way to solve this? >>> >>> # There is insufficient memory for the Java Runtime Environment to >>>> continue. >>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for >>>> committing reserved memory. >>>> >>> >> Use less heap memory? >> >> You haven't specified under which circumstances this occurred, so I can >> only conjecture that it is likely being caused by "writing too fast". >> >> Write more slowly. >> >> =Rob >> >> > >
