Can you post the output of "free" before starting cassandra? Is there anything in logs before it runs out of memory (ie in /var/log/cassandra/output.log)? If an exception is thrown with an OOM it might narrow down if theres a kernel resource (ie mmap file limit, file descriptor limit etc) that you have maxed out of instead of memory.
--- Chris Lohfink On Sep 17, 2014, at 8:35 PM, Yatong Zhang <bluefl...@gmail.com> wrote: > @Chris Lohfink I have 16G memory per node, all the other settings are default > > @J. Ryan Earl I am not sure. I am using the default settings. > > But I've found out it might be because some settings in '/etc/sysctl.conf'. I > am still testing it > > On Wed, Sep 17, 2014 at 11:46 AM, J. Ryan Earl <o...@jryanearl.us> wrote: > Are you using JNA? Did you adjust your memlock limit? > > On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <clohf...@blackbirdit.com> > wrote: > How much memory does your system have? How much memory is system utilizing > before starting Cassandra (use command free)? What are the heap setting it > tries to use? > > Chris > > On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bluefl...@gmail.com> wrote: > >> It's during the startup. I tried to upgrade cassandra from 2.0.7 to 2.0.10, >> but looks like cassandra could not start again. Also I found the following >> log at '/var/log/messages': >> >> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more than >> 120 seconds. >> Sep 16 09:06:59 storage6 kernel: Tainted: G --------------- H >> 2.6.32-431.el6.x86_64 #1 >> Sep 16 09:06:59 storage6 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0 >> 4971 1 0x00000080 >> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082 >> ffffffff81ed4ff0 ffff8803b4f01540 >> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370 >> ffff88042b591ca0 ffff8803b4f01540 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8 >> 000000000000fbc8 ffff8803b4f01af8 >> Sep 16 09:06:59 storage6 kernel: Call Trace: >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? >> exit_robust_list+0x90/0x160 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] >> get_signal_to_deliver+0x1f6/0x460 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ? >> wake_up_new_task+0xd9/0x130 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ? do_fork+0x13d/0x480 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? sys_futex+0x7b/0x170 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] >> do_notify_resume+0x90/0xc0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17 >> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more than >> 120 seconds. >> Sep 16 09:06:59 storage6 kernel: Tainted: G --------------- H >> 2.6.32-431.el6.x86_64 #1 >> Sep 16 09:06:59 storage6 kernel: "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0 >> 4972 1 0x00000080 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082 >> ffffffff81ed6d78 ffff8803b4cf1500 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370 >> ffff8803b4d7fca0 ffff8803b4cf1500 >> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8 >> 000000000000fbc8 ffff8803b4cf1ab8 >> Sep 16 09:06:59 storage6 kernel: Call Trace: >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? >> exit_robust_list+0x90/0x160 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ? >> wake_up_state+0x10/0x20 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] >> get_signal_to_deliver+0x1f6/0x460 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ? >> __switch_to+0x1ac/0x320 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ? >> thread_return+0x4e/0x76e >> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? sys_futex+0x7b/0x170 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] >> do_notify_resume+0x90/0xc0 >> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17 >> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more than >> 120 seconds. >> >> >> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> wrote: >> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bluefl...@gmail.com> wrote: >> I just encountered an error which left a log '/hs_err_pid3013.log'. So is >> there a way to solve this? >> >> # There is insufficient memory for the Java Runtime Environment to continue. >> # Native memory allocation (malloc) failed to allocate 12288 bytes for >> committing reserved memory. >> >> Use less heap memory? >> >> You haven't specified under which circumstances this occurred, so I can only >> conjecture that it is likely being caused by "writing too fast". >> >> Write more slowly. >> >> =Rob >> >> > > >
