On Tue, Oct 2, 2012 at 8:51 AM, Shane McEwan <sh...@mcewan.id.au> wrote:
> Thanks John and Kelly. It's nice to know we're not the only ones. :-)
>
> As I said, we'll be upgrading to 1.2 in the coming weeks so it's good to
> know that the memory issues might go away after that. It's not a showstopper
> for us, more of a curiosity and concern it might develop into something
> worse.
>
> I'll persist with the etop and see if I can get it to run and will report
> back.
>
> We're still using key filters in our MapReduce functions but we plan to move
> to 2i at the same time as upgrading to 1.2.
>
> The word "monitor" doesn't appear in any of our logs for the last 5 days.
> Just lots of:
>
> 2012-10-02 00:10:47.869 [error] <0.31890.1344> gen_fsm <0.31890.1344> in
> state wait_pipeline_shutdown terminated with reason: {sink_died,normal}
> 2012-10-02 00:10:47.909 [error] <0.31890.1344> CRASH REPORT Process
> <0.31890.1344> with 0 neighbours crashed with reason: {sink_died,normal}
> 2012-10-02 00:10:47.981 [error] <0.166.0> Supervisor riak_pipe_builder_sup
> had child undefined started with {riak_pipe_builder,start_link,undefined} at
> <0.31890.1344> exit with reason {sink_died,normal} in context
> child_terminated
>
> Thanks!
>We had this same error before the upgrade. It's much less noisy now but same thing - sink_died > > On 02/10/12 15:55, Kelly McLaughlin wrote: >> >> John and Shane, >> >> I have been looking into some memory issues lately and I would be very >> interested in more >> information about your particular problems. If either of you are able to >> get some output >> from etop using the -sort memory option when you are having elevated >> memory usage it >> would be very helpful to see. I know that sometimes you get the >> connection_lost message >> when trying to use etop, but I have found that sometimes if you keep >> trying it may succeed >> after a few attempts. >> >> Are either of you using MapReduce? I see that John is using 2I. Shane, do >> you also use 2I? >> Finally, do you notice a lot of messages to the console or console log >> that have the either the >> phrase 'monitor large_heap' or 'monitor long_gc'? >> >> Kelly >> >> On Oct 2, 2012, at 6:11 AM, "John E. Vincent" >> <lusis.org+riak-us...@gmail.com> wrote: >> >>> I would highly suggest you upgrade to 1.2 when possible. We were, up >>> until recently, running on 1.4 and seeing the same problems you >>> describe. Take a look at this graph: >>> >>> http://i.imgur.com/0RtsU.png >>> >>> That's just one of our nodes but all of them exhibited the same >>> behavior. The falloffs are where we had to bounce riak. >>> >>> This is what one of our nodes looks like now and has looked like since >>> the upgrade: >>> >>> http://i.imgur.com/pm7Nk.png >>> >>> The change was SO dramatic that I seriously though /stats was broken. >>> I've verified outside of Riak and inside. The memory usage change was >>> very positive. Evidently there's even still a memory leak. >>> >>> We're heavy 2i users. No multi backend. >>> >>> On Tue, Oct 2, 2012 at 4:08 AM, Shane McEwan <sh...@mcewan.id.au> wrote: >>>> >>>> G'day! >>>> >>>> Just recently we've noticed memory usage in our Riak cluster constantly >>>> increasing. >>>> >>>> The memory usage reported by the Riak stats "memory_total" parameter has >>>> been less than 100MB for nearly a year but has recently increased to >>>> over >>>> 1GB. >>>> >>>> If we restart the cluster memory usage usually returns back to what we >>>> would >>>> call "normal" but after a week or so of stability the memory usage >>>> starts >>>> gradually growing again. Sometimes after a growth spurt over a few days >>>> the >>>> memory usage will plateau and be stable again for a week or two and then >>>> put >>>> on another growth spurt. The memory usage starts increasing at the same >>>> moment on all 4 nodes. >>>> >>>> This graph [http://imagebin.org/230614] shows what I mean. The green >>>> shows >>>> the memory usage as reported by "memory_total" (left-hand y-axis scale). >>>> The >>>> red line shows the memory used by Riak's beam.smp process (right-hand >>>> y-axis >>>> scale). >>>> >>>> Also notice that the gradient of the recent growth seems to be >>>> increasing >>>> compared to the memory increases we had in August. >>>> >>>> We might have just assumed that the memory usage was normal Riak >>>> behaviour. >>>> Perhaps we have just tipped over some sort of internal buffer or cache >>>> and >>>> that causes some more memory to be allocated. However, whenever we >>>> notice >>>> the memory usage increasing it always coincides with the "riak-admin >>>> top" >>>> command failing to run. >>>> >>>> We try to run "riak-admin top" to diagnose what is using the memory but >>>> it >>>> returns: "Output server crashed: connection_lost". If we restart the >>>> cluster >>>> the top command works fine (but, of course, there's nothing interesting >>>> to >>>> see after a restart!). >>>> >>>> So our theory at the moment is that some sort of instability or race >>>> condition is causing Riak to start consuming more and more memory. A >>>> side >>>> effect of this instability is that the internal processes needed for >>>> running >>>> the top command are not working correctly. The actual functionality of >>>> Riak >>>> doesn't seem to be affected. Our application is running fine. We see a >>>> slight increase in "FSM Put" times and CPU usage during the memory >>>> growth >>>> phases but all other parameters we're monitoring on the system seem >>>> unaffected. >>>> >>>> There's nothing abnormal in the logs. We get a lot of >>>> "riak_pipe_builder_sup >>>> {sink_died,normal}" messages but they can be ignored, apparently. The >>>> cluster is under constant load so we would expect to see either gradual >>>> memory increase or a steady state but not both. Erlang process count, >>>> open >>>> file handles, etc are stable. >>>> >>>> So I was wondering if anyone has seen similar behaviour before? >>>> Is there anything else we can do to diagnose the problem? >>>> I'm accessing the stats URL once per minute, could that have any side >>>> effects? >>>> We'll be upgrading to Riak 1.2 and new hardware in the next few weeks so >>>> should we just ignore it and hope it goes away? >>>> Any other ideas? >>>> Or is this just normal? >>>> >>>> Riak config: >>>> 4 VMware nodes >>>> ring_creation_size, 256 >>>> n_val, 3 >>>> eleveldb backend: >>>> max_open_files, 20 >>>> cache_size, 15728640 >>>> "riak_kv_version":"1.1.1", >>>> "riak_core_version":"1.1.1", >>>> "stdlib_version":"1.17.4", >>>> "kernel_version":"2.14.4" >>>> Erlang R14B03 (erts-5.8.4) >>>> >>>> Thanks! >>>> >>>> Shane. >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
