I've removed the file from the destination machine and still received the
error. When I disabled the preallocate option it worked. I suspect there
is an issue in the cygwin preallocate option in 1.7. I'll report it to the
cygwin message board.
Rob
--
Please use reply-all for most replies to
The main reason I recommended using vshadow is that you don't have to create
the BKF file so the space used is only what the shadow copy requires (300MB
minimum). You can delete the shadow copy when you are done so no meaningful
space impact on your drive. The shadow copy will also be very fast,
You need to use shadow copies to create a drive you can use with rsync, or
you have to take Exchange offline. You can use vshadow.exe (use version
3.0) to create the shadow copy and then expose it as a drive. Then use
rsync to sync this point-in-time backup to your offsite location across an
ssh
I'm having some issues with some lnk files using cygwin 1.7 and rsync 3.0.3.
I'm not sure if this is an issue with rsync or cygwin.here is the error
written to the rsync log:
rsync: rename "/EDrive/Documents and Settings/csams/Application
Data/Microsoft/Office/Recent/.Order requests Lori Kopp.doc.
ccurs again. You're
probably right, it may be a hardware issue.
Rob
On Tue, 2008-09-30 at 14:20 -0600, Rob Bosch wrote:
> 2008/09/30 12:09:55 [12508] rsync: write failed on "/EDrive/testfile.edb"
> (in Test.Backup):Resource temporarily unavailable (11)
That error is comi
I'm using rsync over an ssh tunnel to transfer some large files. I've been
getting the "Resource temporarily unavailable" in my rsyncd.log frequently.
The connections are T1's and very stable so I don't think it is a network
drop. The disconnect occurs on the same files every time...although thes
I believe I've figured out why the process was taking so long...or at least
have a theory. In the end it appears that much of the data was being sent
even though the "true" amount of data change was less than 7% of the
filesize.
Exchange uses a database page size of 4K. Many times a page is de
> What if you 2>&1 |while read x; do echo "`date`: $x"; done ?
I'm sure this would work great under "real" linux, but I'm running rsync
under cygwin. It is running via a command line using the cygwin1.dll. I
could do it within a bash script but would prefer to keep it in the scripts
I use today.
The two servers are connected via T1 on one end, and 30MB Internet at the
other. Based on the stats I'm looking at the T1 is only about 20% utilized.
The other end is not even close to fully utilized. It would seem to me that
the data connection would be an issue when sending the literal data ver
The files are very similar, a maximum of about 5GB of data differences over
80GB. The CPU on both sides is low (3-5 percent) and the memory usage is
low (11MB on the client, not sure on the server).
Full rsync options are:
-ruityz --partial --partial-dir=.rsync-partial --links --ignore-case
I've been trying to figure out why some large files are taking a long time
to rsync (80GB file). With this file, the match process is taking days.
I've added logging to verbose level 4. The output from match.c is at the
point where it is writing out the "potential match at" message. In a 9 hour
I've been trying to find out how to improve performance with large files
(e.g. Exchange databases > 80GB). One thing I've found is that reviewing
the log output using - is difficult to pinpoint what is taking the
majority of the time without monitoring the log "live" which isn't really
feasibl
I ran an rsync test using the RDC program that comes with the Server 2008
SDK. The RDC program is a simple program to demonstrate Microsoft's Remote
Differential Compression algorithm. It uses the API for RDC to copy a file
across a WAN/LAN connection. The program, as-is, has little functionalit
The maximum value allowed was 128KB. I think this is an issue with cygwin
(using 1.7.6 for testing) and how it interprets an int32. I've modified the
define value from the int32 and put in an upper limit instead to test. Just
an FYI in case someone else has this issue.
Rob
--
Please use rep
I've been trying to test rsync on files over 80GB again to see if I can find
ways to improve performance further. I tried passing a manual block-size
value and I keep getting "Invalid block length" from the [sender]. Can
someone refresh my memory of the valid values that can be passed using this
I've been getting the error below and the only thing I can see odd about the
directory is the ! in the directory name. That and there is a space at the
end of the text which doesn't seem like it should be there either. There
are no permissions issues with accessing the directory and the directory
Wayne, thanks for your help on this issue. It turned out to be a user error
(me) since the client was the pre5 client instead of the pre10. I reran the
test with the pre10 client as you suggested and here are the results. The
only odd thing I noticed is that even though all the data matched, the
Let me know of any additional info or tests you need me to run. I'll halp any
way I can. thanks.
rob
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> A local transfer needs --no-whole-file if you want it to use the rsync
> algorithm (which uses more disk I/O, so it's not the default).
>
>..wayne..
Here are the stats on a small file...exactly as expected with everything
getting matched. Is this an issue with how much of the file rsync can loo
I ran rsync on the 59GB file again without preallocate on XFS. It created
only 383 extents...very low fragmentation.
Rob
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> A local transfer needs --no-whole-file if you want it to use the rsync
> algorithm (which uses more disk I/O, so it's not the default).
>
>..wayne..
I reran this test with the --no-whole-file option and received the exact
same results. Any idea on why some much data is being sent when the files
The patch truncates the file with ftruncate if a transfer fails in
receiver.c. This should avoid the problem you mention. Even if this didn't
occur, the file would exist on the FS with the predefined size. It would be
in the allocation table and exist on the disk (you can see it under Windows
ex
> Was that simply due to writing too-small block to NTFS? In other
> words, would increasing the size of write() calls have fixed it
> instead, without leaving allocated but unused disk space in the case
> of a user-abort with --partial, --partial-dir or --inplace?
It could have been a function o
> Though, did I get the right impression that NTFS generates lots of
> extents for small writes even when nothing else is running?
The fragmentation on NTFS was a problem even when nothing else was running
on the server. The preallocation patch made all the difference on NTFS and
cygwin. In that
> A local transfer needs --no-whole-file if you want it to use the rsync
> algorithm (which uses more disk I/O, so it's not the default).
The transfers occurred across a local network but were on separate machines.
Rob
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinf
> Any idea why Glibc's posix_fallocate makes any difference?
>
> Doesn't it simply write a lot of zeros? In that case, why doesn't
> rsync writing lots of data sequentially result in the same number of
> extents?
The destination server had a lot of other processes running at the same
time. I sus
>For a meaningful test, you should actually write 77GB of data into a new
>file and an ftruncated file and see if there's any difference in the
>resulting fragmentation.
>
>In your patch, you should use fallocate in place of ftruncate. If your
>glibc is like mine and doesn't provide direct access
On Sat, 2008-02-23 at 16:43 -0700, Rob Bosch wrote:
>In your patch, you should use fallocate in place of ftruncate. If your
>glibc is like mine and doesn't provide direct access to fallocate,
>you'll have to use syscall and __NR_fallocate .
I'll run a test with both
e-
From: Jamie Lokier [mailto:[EMAIL PROTECTED]
Sent: Sunday, November 18, 2007 10:06 PM
To: Matt McCutchen
Cc: Rob Bosch; rsync@lists.samba.org
Subject: Re: preallocate CPU usage - pre4
Matt McCutchen wrote:
> I notice that the Linux kernel 2.6.23 has gained a system call
> "fallo
I took a stab at modifying the preallocate.diff patch file replacing it with
ftruncate (attached). Do you think the file looks OK for Linux (obviously
cygwin should use posix_fallocate)? I replaced posix_fallocate with
ftruncate and also removed the check for HAVE_POSIX_FALLOCATE.
Maybe this sh
Just as clarification, when I said the directory does not exist I mean that
is was properly deleted by the server/receiving side. It appears as if
rsync did everything that it should but an error code of zero is being
returned and the exit code is set to 23, non-zero.
Full options:
-ruvvityz --co
I'm getting errors in my rsync.logs using rsync3.0.0pre8 with the -vv
option. The client is running on cygwin as the client, CentOS as the
server. The error in the log is:
rsync: delete_file: rmdir(filepath) failed: No error (0)
When I look on the server/receiving side the directory does not e
Sorry, didn't see the matching if statement. Commenting out the two lines
with ICONV in the config.h fixed my issue. Thanks...
Rob
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
I tried changing the HAVE_ICONV_H and HAVE_ICONV_OPEN but this does not
resolve the issue. I noticed in rsync.c that there is an #endif without a
matching if condition at the end of the iconvbufs function. Should this
function be available if HAVE_ICONV_H is not available? I also don't see
how t
The error on 193 is due to the iconv reference in the while loop.Rob
/usr/src/rsync-3.0.0pre8/rsync.c:85: undefined reference to `_iconv_open'
rsync.o: In function `iconvbufs':/usr/src/rsync-3.0.0pre8/rsync.c:193:
undefined reference to `_iconv'/usr/src/rsync-3.0.0pre8/rsync.c:163: undefined
I've been trying to compile 3.0.0pre8 under cygwin. I was getting a problem
with the iconv functions which I think are related to the cygwin environment
and its iconv.h.
To bypass it I wanted to compile with the --disable-iconv function. I'm still
getting the following errors during compile:
I guess a quick read of the man page would have resolved this. Thanks for
the tip.
Rob
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
I had a case where I was using the --partial option (client is running
rsync-pre5, server daemon is rsync-pre8). The transfer was interrupted and
the partially transferred file replaced the destination file on the server.
My understanding is that the partially transferred file would be kept but it
Just thought I'd post a simple workaround for Windows ACL's and file
attributes for rsync. Using some basic scripting will allow ACL information
to be stored for future use on files transferred using rsync. Most will
probably find the information worthless but thought I'd post it anyway.
- To ge
> Attached is the C++ source for a simple program "allocate" to allocate a
> file. Call the
> program like "./allocate thefile 770".
Now that was fun. I was creating 300GB files with no fragmentation in less
than 1/2 a second! Seriously, the performance using the program was identica
Matt, I have absolutely no programming skills for developing my own program!
I'd be happy to compile and test a program however.
Are you going to put the fix for the preallocate patch in the 3.0 next
pre-release or release?
I'm pretty sure the function is being called properly since I see
When I use the preallocate patch and create a 77GB file using the function I
get a CPU spike on the server-side. The spike lasts about 20 minutes and
uses about 20%-25% of the cpu associated with the rsync instance creating
the file. The spike is directly linked to the time it takes to create the
I'm running pre4 on a 77GB file. It seems like the hash table is taking a
long time to be built. I'm not sure what is involved in this step but as an
example the following is logged during a run:
send_files(11, priv1.edb)
send_files mapped priv1.edb of size 79187419136
calling match_sums priv1
tchen
Sent: Thursday, October 25, 2007 7:03 PM
To: Rob Bosch
Cc: rsync@lists.samba.org
Subject: Re: rsync error: STATUS_ACCESS_VIOLATION
On Thu, 2007-10-25 at 18:55 -0600, Rob Bosch wrote:
> I received the following error on the client when testing the pre2
> release.
> 4 [main] rsyn
I received the following error on the client when testing the pre2 release.
I'm not sure what it means or if it helps. Rsync 2.6.9 runs the same backup
without error. I compiled rsync under cygwin CVS so it may be an issue
there as well. Server side did not crash. It did log an error, but I'm
p
I'm wondering why I'm having trouble getting attributes like Hidden or
System to come across from my Windows rsync under cygwin implementation
(2.6.9). I'm using rsync via an ssh tunnel from one Windows machine to the
other, rsync version 2.6.9. The options I'm using are -ruvitpAE --super
--preal
OK, sounds like completely different scenarios. We haven't been using rsync
locally to do copies or things like that.
-Original Message-
From: Linda Walsh [mailto:[EMAIL PROTECTED]
Sent: Monday, December 18, 2006 2:42 PM
To: Rob Bosch
Cc: 'Matt McCutchen'; 'rsyn
rategies though so please
post what you find out!
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt
McCutchen
Sent: Sunday, December 17, 2006 8:52 AM
To: Linda Walsh
Cc: Rob Bosch; rsync
Subject: Re: File Fragmentation issue...especially relating to NTFS
post the client result. This covers us in there is a drop in the
remote link. Anyway, thanks!
Rob
"Wayne Davison" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> On Mon, Sep 18, 2006 at 11:13:04AM -0600, Rob Bosch wrote:
> > I've noticed that e
Wayne answered in my separate post...ignore this message.
Rob
"Rob Bosch" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> I'm guessing no one else has seen this issue? Most exit codes are
properly
> reported (e.g. code 12, code 10, code 2).
I’m guessing no one else has seen this issue? Most
exit codes are properly reported (e.g. code 12, code 10, code 2). The specific
scenario is when a filename too long or directory not present error causes the client
to exit with a code of 23 (reported as the exit code on the client proper
Normally rsync will provide more information in the output, not just the
error at the end of the log. That is, during the transfer there could be a
line that says "Filename too long" or some other issue with reading the
file. We typically see the 23 error code when either the file name is too
lon
The way I have accomplished this is to modify the rsync request slightly and
cache the modulename and request into a database. In my case, this provides
me with a unique key in the database so I know the requests match up. On
the client side I have added the top-level path into the request (e.g.
I've noticed that even if rsync has a non-zero exit code on the client that
the server still reports an exit status of 0. For example, I've received an
error 23 with the following output on the client-side "rsync error: some
files could not be transferred (code 23) at main.c(954) [sender=2.6.9cvs]
I've looked at these arguments via environment variables and and I can't
find the remote client's pathname in the RSYNC_ARG# values. For example, if
I run the command:
rsync -ruvit --ignore errors /usr/bin host::module/
the ARG values include all the options but the /usr/bin value. I'd like to
Wayne
my vote is for a command-line option. Ive noticed there is some
penalty for very large files (35GB-50GB). The penalty is relatively small
based on my intuitive measurements
read me watching without running a real
timer. The difference is very small compared to what happens after a few
w
Wayne…my
vote is for a command-line option. I’ve noticed there is some penalty
for very large files (35GB-50GB). The penalty is relatively small based on my ‘intuitive’
measurements…read me watching without running a real timer. The
difference is very small compared to what happens after
That would be great! If you need any help testing, let me know.
Rob
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt
McCutchen
Sent: Friday, August 25, 2006 6:15 PM
To: Rob Bosch
Cc: rsync
Subject: Re: File fragmentation
On 8/25/06, Rob Bosch
alf Of Matt
McCutchen
Sent: Monday, August 14, 2006 5:33 PM
To: Rob Bosch
Cc: rsync@lists.samba.org
Subject: Re: File fragmentation
On 8/13/06, Rob Bosch <[EMAIL PROTECTED]> wrote:
> Is this something that should be implemented at the rsync level or cygwin
> level? It would seem th
PROTECTED] On Behalf Of Matt
McCutchen
Sent: Thursday, August 10, 2006 6:59 PM
To: Rob Bosch
Cc: rsync@lists.samba.org
Subject: Re: File fragmentation
On 8/6/06, Rob Bosch <[EMAIL PROTECTED]> wrote:
> I was wondering why rsync doesn't preallocate the space like copy
> or othe
Testing with an updated version of cygwin
(1.5.21) proved to resolve the problem. Thanks Tev!!!
Rob
From: Tevfik Karagülle
[mailto:[EMAIL PROTECTED]
Sent: Monday, August 07, 2006 3:36
PM
To: 'Rob Bosch';
rsync@lists.samba.org
Subject: RE: Invalid File I
Good plan…I’ll test that now
and post my results.
From: Tevfik Karagülle
[mailto:[EMAIL PROTECTED]
Sent: Monday, August 07, 2006 3:36
PM
To: 'Rob Bosch';
rsync@lists.samba.org
Subject: RE: Invalid File Index
error
cwRsync version 2.0.8
uses newer versions
Since upgrading to
2.6.8 with the tag 3 patch, I’ve been consistently getting an error using
rsync. We have about 100 servers using the client to a central backup
server and the issue is repeatable on the servers having the problem. About
10 of the servers are having the problem. The err
I've been running
some tests on files created by rsync and noticing fragmentation issues. I
started the testing because our 5TB array started performing very slowly and it
appears fragmentation was the culprit. The test I conducted was straighforward:
1. Copy over a 49GB file. Analyzed wit
64 matches
Mail list logo
