Hi Gordon, Thank you for sharing the information. We are seeing the same exact type of behavior from our search cluster. I have tracked the problem(s) though the query system. It looks like the mailboxes we are both seeing are "abandoned" and / or the messages are never matched within the Erlang code (it_op_collector_loop, riak_search_op_utils.erl); the messages are then never processed, therefore the resources they utilize never released. This is a major problem.
I have been debugging this for some time and I wish I could say it was going well. The implementation is convoluted -- have you gotten through it? Can you verify the same cause? We have been internally discussing the possibility of removing this query processing implementation completely and replacing it with something built in-house because the problems we have uncovered trying to debug the "abandoned mailbox" problem are related and systemic: 1) indeterminate and possibly very large data structures created and manipulated for intermediate and final sets of results, 2) very poor or non-existent ability to gain any insight into what is executing within the "plumbing" of the current query execution system without "herculean" effort (in my opinion), and 3) unacceptable performance (predictably or subjectively) from the merge_index riak_search backend. Are there any other backends available for riak_search with the Enterprise Riak offering? I really like the design of riak_search but the performance seems to be only a very small fraction of our equivalent SOLR installation, even with several times the amount of resources "thrown at it" -- it does not seem to use resources we "throw at it" well, either, or in the mailboxes case, responsibly. I will quickly admit I may be doing something wrong. Is there a user-error situation in which mailboxes should be abandoned taking up memory? Does anyone else have experiences with equivalent riak_search vs. SOLR installations? Thanks again for sharing Gordon. Your results make me feel like this may not be entirely stupidity on my part. Gilbert On Tue, May 31, 2011 at 8:51 AM, Gordon Tillman <gtill...@mezeo.com> wrote: > Howdy Gilbert, > I reproduced the issue this morning and then ran the command that you > specified on two of the non-empty mailboxes. > The output from that is posted here: > https://gist.github.com/1000735 > Please let me know if this corresponds to the issue that you are seeing. > Thank you, > --gordon > > On May 27, 2011, at 20:10 , Gilbert Glåns wrote: > > Gordon, > Could you try: > > erlang:process_info(list_to_pid("<0.16614.32>"), [messages, > current_function, initial_call, links, memory, status]). > > in a riak search console for one/some of those mailboxes and share the > results? I am curious to see if you are having the same systemic > memory consumption I am experiencing. > > Gilbert > > On Fri, May 27, 2011 at 5:15 PM, Gordon Tillman <gtill...@mezeo.com> wrote: > > Howdy Gang, > > We are having a bit of an issue with our 3-node riaksearch cluster. What is > happing is this: > > Cluster is up and running. We start testing our application against it. As > the application runs the erlang process consumes more and more memory > without ever releasing it. > > In trying to investigate the issue we ran the riaksearch-admin cluster_info > command. It appears that the bulk of this memory is being consumed by a > bunch of mailboxes. > > I have posted both the output of the cluster_info command and the app.config > from one of the nodes here: > > https://gist.github.com/996419 > > I would be very grateful if someone from Basho would take a look at the > cluster_info and see if they can spot anything obvious. > > Each machine in the cluster has an 8-core Xeon and 16GB RAM. I believe all > of the platform details, etc., are in the cluster_info dump. > > Many thanks, > > --gordon > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
