Hi Paul,
Thanks for your inputs. I think I've got an idea of whats taking place.
Please see if what I have mentioned makes sense.
The reason file transfer is taking a few secs in the partial transfer case
is cause the file is highly compressible and rsync
is using that to transfer less bytes to replicate the file. The
file contains lines of chars. The same char is repeated on each line.
And the pattern is repeated. I yanked multiple lines and posted them to
increase the file size. My bad.. I used a wrong test file..
When I tarred the file the size is just about 197 bytes.. Will try
transferring/syncing a tarred file to start with. Its probably the
uncompressing and replication of the file that increased the CPU usage of
rsync on the client to 80-90% which I though was
due to increased transfer rate.
The tarred file size
---------------------------
ls -la foo.txt.tar.gz
-rw-r--r-- 1 root root 197 Jan 22 15:40 foo.txt.tar.gz
I am taking note of time on a watch/PC. I am syncing a file of size
6,615,128 bytes (~6.3MB) which should take about
646 secs to transfer at 10KBps. When the file is transferred completely then
it does take a little less than 11 mins to
complete.
Test 1: transfer complete file
The rsync server and client logs also show that the time taken is
little less than 11 mins when the complete file has to be transferred.
rsync server daemon logs which is running on the cloud
------------------------------------------------------------------------------
2011/01/22 14:32:13 [12454] forward name lookup for YYYYYY failed: No
address associated with hostname
2011/01/22 14:32:13 [12454] connect from UNKNOWN (XXX.XXX.XXX.XXX)
2011/01/22 09:02:14 [12454] rsync on sessions/example-7-step-1/* from
UNKNOWN (XXX.XXX.XXX.XXX)
2011/01/22 09:02:15 [12454] building file list
2011/01/22 09:13:02 [12454] sent 6,616,898 bytes received 583 bytes total
size 6,615,128
--------------------------------------------------------------------------------------------------------------------------------------------
rsync client logs which is running on my PC
------------------------------------------------------------------------------
receiving incremental file list
data/
data/foo.txt
meta/
Number of files: 3 (reg: 1, dir: 2)
Number of created files: 3 (reg: 1, dir: 2)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 6,615,128 bytes
Total transferred file size: 6,615,128 bytes
Literal data: 6,615,128 bytes
Matched data: 0 bytes
File list size: 80
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 578
Total bytes received: 6,616,878
sent 578 bytes received 6,616,878 bytes 10,188.54 bytes/sec
total size is 6,615,128 speedup is 1.00
-------------------------------------------------------------------------------
Test 2: Start with transferring complete file i.e. remove the file on the
client from the previous run.
Midway I remove the ethernet cable or kill the rsync client.
The partial transfer is of size 655360 bytes. i.e. 5959768 bytes
are pending
The remaining transfer should take around 582 seconds but when I
start the rsync client again
it just takes about 4 to 6 seconds.
rsync server daemon logs which is running on the cloud
------------------------------------------------------------------------------
2011/01/22 15:05:19 [12615] forward name lookup for XXXXXX failed: No
address associated with hostname
2011/01/22 15:05:19 [12615] connect from UNKNOWN (XXX.XXX.XXX.XXX)
2011/01/22 09:35:20 [12615] rsync on sessions/example-7-step-1/* from
UNKNOWN (XXX.XXX.XXX.XXX)
2011/01/22 09:35:20 [12615] building file list
2011/01/22 09:36:33 [12615] rsync: [sender] write error: Broken pipe (32)
2011/01/22 09:36:33 [12615] rsync: connection unexpectedly closed (109 bytes
received so far) [sender]
2011/01/22 09:36:33 [12615] rsync error: received SIGINT, SIGTERM, or SIGHUP
(code 20) at io.c(-1604) [sender=3.1.0dev]
2011/01/22 15:07:54 [12624] forward name lookup for XXXXXX failed: No
address associated with hostname
2011/01/22 15:07:54 [12624] connect from UNKNOWN (XXX.XXX.XXX.XXX)
2011/01/22 09:37:54 [12624] rsync on sessions/example-7-step-1/* from
UNKNOWN (XXX.XXX.XXX.XXX)
2011/01/22 09:37:55 [12624] building file list
2011/01/22 09:38:02 [12624] sent 39,793 bytes received 4,938 bytes total
size 6,615,128
check size of partial data transfer after killing the rsync client/removing
the ethernet cable
------------------------------------------------------------------------------------------------------------------------------
ls -la rsync-partial/foo.txt
-rw-r--r-- 1 root root 655360 Jan 22 15:06 rsync-partial/foo.txt
------------------------------------------------------------------------------------------------------------------------------
rsync client logs which is running on my PC
------------------------------------------------------------------------------
receiving incremental file list
data/
data/foo.txt
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(611)
[generator=3.1.0dev]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(611)
[receiver=3.1.0dev]
rsync: [receiver] write error: Broken pipe (32)
receiving incremental file list
data/foo.txt
Number of files: 3 (reg: 1, dir: 2)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 6,615,128 bytes
Total transferred file size: 6,615,128 bytes
Literal data: 6,496 bytes
Matched data: 6,608,632 bytes
File list size: 80
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4,933
Total bytes received: 39,773
sent 4,933 bytes received 39,773 bytes 5,259.53 bytes/sec
total size is 6,615,128 speedup is 147.97
------------------------------------------------------------------------------------------------------------------------------
Regards,
Alistair
On Fri, Jan 21, 2011 at 6:45 PM, Paul Slootman
<paul+rs...@wurtel.net<paul%2brs...@wurtel.net>
> wrote:
> On Fri 21 Jan 2011, Alistair Dsouza wrote:
> >
> > I have set the bwlimit to 10 i.e. 10 KBps. When I try to sync files the
> > bwlimit is as what is
> > set. I have enabled storage of partial files. When a communication break
> > causes rsync to
> > time out during a transfer the partial file is stored in the partial
> > directory. I then try rsync
> > again when the link is up. At this point I see that the bwlimit of 10KBps
> is
> > not followed.
> > rsync transfers the data as fast as possible.
>
> How did you measure this data rate?
Are you using the --progress output? Then you need to realize that what
> is shown there is not the data rate over the network, but the effective
> rate at which the file is transferred. As the first part of the file is
> in the partial directory, the data that needs to be transmitted is not
> much more than the block checksums plus a little overhead. So then it
> looks like the file is being transmitted a lot faster.
>
> The same thing can be seen when using -z while transferring a highly
> compressable file: --progress will show a much higher transfer rate than
> what you've configured with --bwlimit.
>
> This phenomenon can be easily verified by doing it over a dialup line,
> or other medium that physically can't go faster than a given rate:
> --progress will show higher rates than would be possible.
>
>
>
> Paul
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
