The biggest problem we've found with modern day gateways is that they are dramatically overbuffered and this messes with tcp's congestion avoidance mode, a lot.
Innumerable resources on "bufferbloat" are on bufferbloat.net to this effect: https://www.bufferbloat.net/projects/bloat/wiki/TechnicalIntro and for various rants and data: http://gettys.wordpress.com/ You can mitigate your problem with an fq_codel based shaper on the gateway with something like CeroWrt's "SQM" system, (or openwrt's or dd-wrts, etc) http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310 and further make rsync invisible by marking it with the diffserv marking of CS1 in that system. I submitted patches long ago to add classification and support for other tcps' congestion control algorithms like lp and westwood to rsync's patch db. Info on codel and fq_codel: https://datatracker.ietf.org/doc/draft-nichols-tsvwg-codel/ https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00 On Fri, Apr 4, 2014 at 10:28 AM, Satish Shukla <sat...@cadence.com> wrote: > With multiple rsync (I run around 10) streams the bwlimit becomes > complicated w.r.t. optimizing total bandwidth, its no way near close to > what I want to achieve. I want it to be dynamically scale up and down > within maximum threshold depending on overall load. trickle ( > http://monkey.org/~marius/pages/?page=trickle) seems to be promising but > I haven't tried it yet. > > > ~satish > > > On 4/4/2014 1:03 PM, Frank Terhaar-Yonkers wrote: > > > I needed to back up some of my NAS over the WAN to another (friends) NAS > (3/4TB-total). A lot of expensive hi-def music files, mostly very large. > He backs up to mine, vice versa for disaster recovery. > > There were two issues: 1) sucking up all my UL bandwidth, only 5mbit in > this case, 2) having the option to do a "delayed" kill of the rsync when it > *finished* the current file rather than an immediate stop so as not to > waste the bandwidth already used to move a portion of a large file. > > I implemented both with signals. A signal toggles the enable/disable of > the --bwlimit=XXX param so you can do a simple throttle back or run wide > open. The second signal is a delayed quit - when current file is finished. > > This seemed to work pretty well. During the day when I needed the BW for > work, etc. I'd throttle it, then let it run wide open at night. This > could be expanded with a BW list, so instead of a simple toggle, the next > BW setting in the list could be used, round-robin. The delayed quit was > nice if new content was added that I felt I wanted backed up first. Quit > (delayed), then restart, easily scripted. > > Let me know if there is any interest in putting this in the code base and > I'll create some diffs for review. > > Cheers, > Frank > > On 4/3/14, 10:47 AM, Marian Marinov wrote: > > On 04/03/2014 03:35 PM, Christoph Biedl wrote: > > Joe wrote... > > This is way beyond my level of expertise, but wouldn't something like > ionice help with that? > > > Although I'm not Marian, probably not. The ionice program does a > reasonable good job when it's about prioritizing read operation. The > context makes me guess it's rather about writing. > > > We were using ionice and it did not help. The problem is that if for some > reason all of your concurrent rsyncs are running with the same ionice level > and class, even thou they are ioniced, effectively there is no change. > So at that point we wrote a simple perl daemon that monitored rsyncs and > changed ionice levels based on the amount of time each rsync spent in each > level. > We pushed all new rsyncs onto the lowest level and dynamically moved them > to the top. It simply does not work as well as I imagined. > > The best solution was to use the bklio control-group, BUT, with it the > backups were still slower then with the slow-down option. > > Marian > > > Also, check out: > > 2 more pipe utilities > > Viewer & throttle > http://www.ivarch.com/programs/pv.shtml > > Throttle - limits bandwidth of a pipe - for use with network transfers > http://linux.die.net/man/1/throttle > > > The pv utility is way to little known and served me well in many > situations, and throttle seems to do quite the same. Both however seem > to do quite the same thing rsync's --bwlimit option does, while the > latter is more sophisticated. > > It the issue is the one I have in mind (which is one I constantly > suffer from), the actual problem is the dirty buffer writeback > strategy, deep in the Linux kernel. > > Christoph > > > > -- > ------------------------------ > > Frank Terhaar-Yonkers W4FTY > Cisco Systems, Inc. > 7025 Kit Creek Road PO Box 14987 > Research Triangle Park, North Carolina 27709 > f...@cisco.com voice(919)392-2101 > > > *JabberCall me* > <https://sjc-jabberc-ext.cisco.com/call/83922...@cisco.com?name=RTP%20Phone> > browser-based > video chat > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Dave Täht NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
