With multiple rsync (I run around 10) streams the bwlimit becomes
complicated w.r.t. optimizing total bandwidth, its no way near close to
what I want to achieve. I want it to be dynamically scale up and down
within maximum threshold depending on overall load. trickle
(http://monkey.org/~marius/pages/?page=trickle) seems to be promising
but I haven't tried it yet.
~satish
On 4/4/2014 1:03 PM, Frank Terhaar-Yonkers wrote:
I needed to back up some of my NAS over the WAN to another (friends)
NAS (3/4TB-total). A lot of expensive hi-def music files, mostly very
large. He backs up to mine, vice versa for disaster recovery.
There were two issues: 1) sucking up all my UL bandwidth, only 5mbit
in this case, 2) having the option to do a "delayed" kill of the rsync
when it *finished* the current file rather than an immediate stop so
as not to waste the bandwidth already used to move a portion of a
large file.
I implemented both with signals. A signal toggles the enable/disable
of the --bwlimit=XXX param so you can do a simple throttle back or run
wide open. The second signal is a delayed quit - when current file is
finished.
This seemed to work pretty well. During the day when I needed the BW
for work, etc. I'd throttle it, then let it run wide open at night.
This could be expanded with a BW list, so instead of a simple toggle,
the next BW setting in the list could be used, round-robin. The
delayed quit was nice if new content was added that I felt I wanted
backed up first. Quit (delayed), then restart, easily scripted.
Let me know if there is any interest in putting this in the code base
and I'll create some diffs for review.
Cheers,
Frank
On 4/3/14, 10:47 AM, Marian Marinov wrote:
On 04/03/2014 03:35 PM, Christoph Biedl wrote:
Joe wrote...
This is way beyond my level of expertise, but wouldn't something like
ionice help with that?
Although I'm not Marian, probably not. The ionice program does a
reasonable good job when it's about prioritizing read operation. The
context makes me guess it's rather about writing.
We were using ionice and it did not help. The problem is that if for
some reason all of your concurrent rsyncs are running with the same
ionice level and class, even thou they are ioniced, effectively there
is no change.
So at that point we wrote a simple perl daemon that monitored rsyncs
and changed ionice levels based on the amount of time each rsync
spent in each level.
We pushed all new rsyncs onto the lowest level and dynamically moved
them to the top. It simply does not work as well as I imagined.
The best solution was to use the bklio control-group, BUT, with it
the backups were still slower then with the slow-down option.
Marian
Also, check out:
2 more pipe utilities
Viewer & throttle
http://www.ivarch.com/programs/pv.shtml
Throttle - limits bandwidth of a pipe - for use with network transfers
http://linux.die.net/man/1/throttle
The pv utility is way to little known and served me well in many
situations, and throttle seems to do quite the same. Both however seem
to do quite the same thing rsync's --bwlimit option does, while the
latter is more sophisticated.
It the issue is the one I have in mind (which is one I constantly
suffer from), the actual problem is the dirty buffer writeback
strategy, deep in the Linux kernel.
Christoph
--
------------------------------------------------------------------------
Frank Terhaar-Yonkers W4FTY
Cisco Systems, Inc.
7025 Kit Creek Road PO Box 14987
Research Triangle Park, North Carolina 27709
f...@cisco.com voice(919)392-2101
*JabberCall me*
<https://sjc-jabberc-ext.cisco.com/call/83922...@cisco.com?name=RTP%20Phone>
browser-based video chat
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
