Thanks Clay, very helpful!
On 14/12/16 13:06, Clay Gerrard wrote:
I strongly prefer to configure rsyncd with a module per disk:
https://github.com/openstack/swift/blob/c0640f87107d84d262c20bdc1250b805ae8f9482/etc/rsyncd.conf-sample#L25
and then tune the per-disk connection limit to 2-4
There's not really hard and fast rule, in some sense it's related to
replicator concurrency (the producer of the rsync's that *want* those
connections) - targeting roughly:
replicator-concurrency * #nodes ~== (max_conenction * nodes or
max_connection_per_module_per_disk * #disks)
But max connections is about enforcing a sane limit, if you hit that
limit *a lot* it may be limiting throughput (and can lead to other
inefficiencies). But having a few max connections fire now and again
is probably a good thing, since there's no guarantee that every
replicator coro in the cluster might not be thinking it should be
talking to a partition that just so happen to all have a replica on
this one disk at this one time. You generally don't want too many
connections hammering a single spindle because of the await it
introduces for client requests that might need to hit that disk (but
there's auditor configuration management as well, and the new ionice
options for tuning might be an option as well).
-Clay
On Tue, Dec 13, 2016 at 3:11 PM, Mark Kirkwood
<mark.kirkw...@catalyst.net.nz <mailto:mark.kirkw...@catalyst.net.nz>>
wrote:
Any thoughts on this one guys?
On 06/12/16 09:56, Mark Kirkwood wrote:
Hi,
Is there a way to calculate rsync max connection (i.e based on
number of hosts, devices, object server workers etc)?
Some context: we have recently gone live with Swift 2.7.0
cluster (2 regions, 6 hosts overall, each with 16 cores/32
threads and 4 drives). We have rsync max connections to 25
(which I think is being set by puppet-swift.., is the default
value). We are seeing errors in the rsync log: 'max
connections (25) reached'. I'd like to be able to say to the
operations guys: set 'max connections to x, because <some
rules for calculating it>'.
Cheers
Mark
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
Post to : openstack@lists.openstack.org
<mailto:openstack@lists.openstack.org>
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
Post to : openstack@lists.openstack.org
<mailto:openstack@lists.openstack.org>
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
