Thank you for the suggestion Chris. I will try increasing zdbbl, but it
seems like we might have to move to Riak CS.
-Bora
On Wed, Mar 4, 2015 at 12:30 PM, Christopher Meiklejohn <
cmeiklej...@basho.com> wrote:
>
> > On Mar 4, 2015, at 2:22 PM, Bora Kou <b...@zendesk.com> wrote:
> >
> > Hi,
> >
> > We have a 6 nodes riak cluster configured with 256 size. We are using
> versions 1.4.8. Occasionally, we see the server return 503 response.
> Looking at the log, it seems that we have a lot of warning about reading
> large object and busy_dist_port. The majority of the objects are less than
> 2MB but we do have some large objects in the ranch of 5-15MB.
> >
> >
> > These are the two type of log entries we consistently see across all
> nodes when that happen:
> >
> > 2015-03-04 17:53:48.149 [warning]
> <0.1142.0>@riak_kv_vnode:do_get_object:1299 Reading large object of size
> 5498696 from...
> >
> > 2015-03-04 19:06:35.824 [info]
> <0.95.0>@riak_core_sysmon_handler:handle_event:92 monitor busy_dist_port
> <0.32259.1162>
> [{initial_call,{riak_kv_get_fsm,init,1}},{almost_current_function,{gen_fsm,loop,7}},{message_queue_len,0}]
> {#Port<0.3353>,'riak@servername...'}
> >
> > The doc suggest increasing +zdbbl to deal with busy_dist_port. But does
> anybody have any other suggestion to what other parameters I should look at
> to improve the situation?
>
> Hi Bora,
>
> Given that most of the internode communication of Riak is performed over
> Distributed Erlang, which uses a single TCP connection between nodes,
> storing large objects [3] can create throughput problems in the cluster
> such as head of line blocking [1] and filling up the Erlang’s bounded
> distribution buffers (messages are stored here waiting to be sent over the
> TCP connection; any message attempting to perform a TCP send will
> synchronize on this buffer) [2].
>
> Increasing zdbbl increases the size of this buffer, which, if you’re
> system has the available memory is a smart choice. Carefully balance
> increasing the distribution buffer size with available system memory, given
> you don’t want to overflow into virtual memory, which will further slow
> down your system.
>
> If you’re going to be storing objects greater than 1 MB, we recommend
> using Riak CS.
>
> - Christopher
>
> [1] http://en.wikipedia.org/wiki/Head-of-line_blocking
> [2]
> http://docs.basho.com/riak/latest/community/faqs/logs/#riak-logs-have-busy_dist_port-messages
> [3]
> http://stackoverflow.com/questions/24389183/riak-returns-an-error-reading-large-object-of-size-when-a-mapreduce
>
> Christopher Meiklejohn
> Senior Software Engineer
> Basho Technologies, Inc.
> cmeiklej...@basho.com
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
