Well the local test was just to a test to see if I could understand why
the remote sync of the exchange database was so slow. I've heard that
rsync is less efficient for local copies but this isn't like 80% the
performance, or half the performance.. It's a massive difference - which
I wasn't expecting to see.
I have looked at unison but I also backup several linux and bsd systems
here and rsync just seemed like a good choice since it's still be
developed and it works on everything.
In the end, I'm still looking to understand why a sync of a large
exchange database file is taking 30-40 hours to finish and if there's
anything I can do to help reduce that window. I can't add a cache drive
to the FreeBSD server very easily at the moment so I was trying to
narrow down if the issue is on the BSD side or Windows side or maybe a
mix of both.
I wish a native windows client of rsync existed :)
------ Original Message ------
From: "Kevin Korb" <k...@sanitarium.net>
To: rsync@lists.samba.org
Sent: 2/10/2014 4:09:15 PM
Subject: Re: Rsync performance with large exchange database files
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rsync is known to be pretty inefficient on local copies (-W is forced
there btw) and cygwin doesn't really help with that either.
Essentially, when not networking rsync isn't much smarter than cp but
it has a ton of extra overhead.
Also, maybe you want unison since there is a native Windows version of
it?
On 02/10/2014 05:05 PM, br...@sqls.net wrote:
Clean copy. I even used the -W flag to see if it made a difference
but, nope.
I'm testing this same test on some of my other servers too. See if
there's any common-ground I can find.
On another servers (MS SQL Server) with faster disks I tried a
similar test just now. There's only the C drive on this server but
I used my same test file from the other server and used rsync
(3.1.0) to copy the folder from one folder to another folder and it
kicked off and got up to about 25MB/sec. Thing is if I just use
windows to copy the same file from one folder to the other it does
the whole file (3.7GB) in about 5.5 seconds (timed with my phone)
so that's also a pretty massive difference.
Maybe this is normal and I've just not noticed it on these other
servers since they have a much smaller amount of data to backup?
Still seems like some thing is wrong. I wouldn't expect the speed
difference to be that huge.
------ Original Message ------ From: "Cary Lewis"
<cary.le...@gmail.com> To: br...@sqls.net Sent: 2/10/2014 3:56:35
PM Subject: Re: Rsync performance with large exchange database
files
when you were doing rsync from /cygdrive/c to /cygdrive/d was
the exchange file already there? Or was it clean copy?
On Mon, Feb 10, 2014 at 4:03 PM, <br...@sqls.net> wrote:
Okay, so I've done some testing..
I created a roughly 4gb file from one of the smaller exchange
database files.
If I copy that to remotely to my desktop, I get about
45-50MB/sec read speed off the D (exchange database) drive. If
I copy that back to the C drive (just the OS) for the Windows
server it writes to the C drive at almost 100MB/sec over the
network.
If I copy directly from the server D drive to it's C drive
using windows it's around 45MB/sec
Inside cygwin using just the copy command I get about
35-45MB/sec transfer speed so there's a little hit just from
cygwin.
Using rsync to "sync" the file from the D drive to the C drive
with the --progress option. I'm getting about 2-2.5MB/sec
transfer speed
The server is being used... So I've run the tests a few times
thoughout the last hour or so and these are about my average
numbers.
Why would rsync be so much slower? Is there something I can
test to help figure this out? I'm using rsync on a couple dozen
Windows servers and it's been working great so I'm not sure why
this one is acting weird.
------ Original Message ------ From: "Kevin Korb"
<k...@sanitarium.net> To: rsync@lists.samba.org Sent: 2/10/2014
10:57:08 AM Subject: Re: Rsync performance with large exchange
database files
3.1.0 will probably help some.
What are the specs of the FreeBSD system? I have found that ZFS on
FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is
the minimum if dedup is disabled and 16BG of RAM for when dedup is
enabled.
Also, a cache disk helps a lot.
On 02/10/2014 10:22 AM, br...@sqls.net wrote:
------ Original Message ------ From: br...@sqls.net
<mailto:br...@sqls.net> To: rsync@lists.samba.org
<mailto:rsync@lists.samba.org> Sent: 2/10/2014 8:38:06 AM
Subject: Rsync performance with large exchange database
files
I'm using a mixture of FreeBSD w/ ZFS+snapshots and
rsync to backup all the servers at my day job. This
works pretty good overall but on one server it's not
working so well :)
We have an Exchange 2003 server with 4 separate mail
store databases. One of them is roughly 900GB the
others are ~200GB, ~160GB, and ~50GB. Rsync seems to
spend a lot of time trying to find the differences in
the files. On the Windows server where rsync is kicked
off there's very little CPU or RAM usage for the rsync
client. On the server rsync (rsyncd, no ssh) is using
around 70-85% of a cpu (well, half a cpu due to hyper
threading). I'm using VSS on the windows server to take
a snapshot and expose it then running rsync from that
to avoid locking issues.
Is there anything I should check to help narrow down
"problems?" or any settings I should try that could
help speed things up any?
Below is the final output of the last two rsync runs to
give you an idea. It's taking 30-40+ hours to finish
even though it's only transferring 80-160GB of change.
Right now I'm testing this against a local rsync server
so it should get pretty fast network performance.
Eventually it will be moved to our off-site backup but
that connection is still pretty fast (20 MBbit) and
the backup is only hitting 800-1000 Kbytes/sec.
Number of files: 19 Number of files transferred: 6
Total file size: 1265.74G bytes Total transferred file
size: 1057.06G bytes Literal data: 160.67G bytes
Matched data: 896.39G bytes File list size: 482 File
list generation time: 0.001 seconds File list transfer
time: 0.000 seconds Total bytes sent: 160.71G Total
bytes received: 73.74M
sent 160.71G bytes received 73.74M bytes 991.84K
bytes/sec total size is 1265.74G speedup is 7.87
[sender] _exit_cleanup(code=0,
file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c,
line=1052): about to call exit(0)
real 2833m1.324s user 2225m55.906s sys 45m10.015s
Number of files: 11 Number of files transferred: 6
Total file size: 1268.78G bytes Total transferred file
size: 1251.04G bytes Literal data: 83.43G bytes Matched
data: 1167.61G bytes File list size: 216 File list
generation time: 1.360 seconds File list transfer time:
0.000 seconds Total bytes sent: 83.48G Total bytes
received: 87.25M
sent 83.48G bytes received 87.25M bytes 836.85K
bytes/sec total size is 1268.78G speedup is 15.18
real 1745m5.647s user 1129m14.000s sys 39m58.875s
Thanks (in advance) for the help :)
rsync options I'm using on the client are : -rltihv
--progress --stats --inplace --modify-window=1
On the windows client I'm using cygwin + rsync 3.0.9 but
I'm going to test 3.1.0 there and see if there's a
difference.
On the server it's rsync 3.1.0 running rsyncd.
Perhaps useful bit of information :).
-- Please use reply-all for most replies to avoid omitting
the mailing list. To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync Before
posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
-- Please use reply-all for most replies to avoid omitting the
mailing list. To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync Before posting,
read: http://www.catb.org/~esr/faqs/smart-questions.html
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlL5TgsACgkQVKC1jlbQAQfNTwCfUV1iYYTujpQLHQxdBD1KFLm8
N9kAoMlhdECrmIT/LwC0tUHgXcYMjDfI
=42pY
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing
list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
