Hi Nicolas, This is expected behavior currently, but a sync fairness mechanism that will permit load-sharing across gateways during replication is being worked on.
regards, Matt On Fri, Oct 9, 2020 at 6:30 AM Nicolas Moal <nicolas.m...@ubisoft.com> wrote: > > Hello Paul, > > Thank you very much for pointing us at BBR ! We will definitely run some > tests before and with the change applied to see if it's increasing a bit our > transfer speed. > > One additional question if you don't mind. As of today, our zonegroup > configuration consist of two zones, a master zone with a HAProxy VIP, and a > slave zone with another HAProxy VIP. Those VIPs are acting as endpoints for > users for accessing the cluster but are also used for the replication traffic > between the master and the slave zone, so it's one frontend IP with the > RadosGWs configured as backends in the HAProxy configuration for both. > > Let's say we configure multiples VIPs as endpoints in the zonegroup > configuration, will Ceph load-balance the traffic between those endpoints for > the replication traffic or take advantages of the multiples endpoints to > multithread the replication traffic between those endpoints, thus increasing > the overall replication speed ? > > Our initial tests show that no matter how many endpoints you specify in the > configuration, it will only use one source IP and destination IP at a time > and fuel the replication traffic throught this one, and only this one. > > Is it the expected behavior or we are missing something ? > > Thanks again ! > > Cheers, > > Nicolas > > Téléchargez Outlook pour iOS<https://aka.ms/o0ukef> > ________________________________ > De : Paul Mezzanini <pfm...@rit.edu> > Envoyé : Thursday, October 8, 2020 6:51:36 PM > À : Nicolas Moal <nicolas.m...@ubisoft.com>; ceph-users <ceph-users@ceph.io> > Objet : Re: Multisite replication speed > > With a long distance link I would definitely look into switching to BBR for > your congestion control as your first step. > > Well, your _first_ step is to do an iperf and establish a baseline.... > > A quick search and this link seems to explain it not-too-bad > https://www.cyberciti.biz/cloud-computing/increase-your-linux-server-internet-speed-with-tcp-bbr-congestion-control/ > > We have used it before with great success for long distance, high throughput > transfers. > > -paul > -- > Paul Mezzanini > Sr Systems Administrator / Engineer, Research Computing > Information & Technology Services > Finance & Administration > Rochester Institute of Technology > o:(585) 475-3245 | pfm...@rit.edu > > CONFIDENTIALITY NOTE: The information transmitted, including attachments, is > intended only for the person(s) or entity to which it is addressed and may > contain confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon this > information by persons or entities other than the intended recipient is > prohibited. If you received this in error, please contact the sender and > destroy any copies of this information. > ------------------------ > > ________________________________________ > From: Nicolas Moal <nicolas.m...@ubisoft.com> > Sent: Thursday, October 8, 2020 10:36 AM > To: ceph-users > Subject: [ceph-users] Multisite replication speed > > Hello everybody, > > We have two Ceph object clusters replicating over a very long-distance WAN > link. Our version of Ceph is 14.2.10. > Currently, replication speed seems to be capped around 70 MiB/s even if > there's a 10Gb WAN link between the two clusters. > The clusters themselves don't seem to suffer from any performance issue. > > The replication traffic leverages HAProxy VIPs, which means there's a single > endpoint (the HAProxy VIP) in the multisite replication configuration. > > So, my questions are: > - Is it possible to improve replication speed by adding more endpoints in the > multisite replication configuration? The issue we are facing is that the > secondary cluster is way behind the master cluster because of the relatively > slow speed. > - Is there anything else I can do to optimize replication speed ? > > Thanks for your comments ! > > Nicolas > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
