ffic transmitted.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 069
ut I do think it
should be safe.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 0690
t definition in favor of one defining it to 0 - e.g.:
/* #define ARENA_SIZE (32 * 1024) */
#define ARENA_SIZE 0
The arena handling seems to be reasonably tight, so it's probably a
long shot in any event.
-- David
/----
out that 1.6%.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902
ode
that it be done that way - the sequence was pretty much
generate/send/free.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 7
se in our checksum time - although this
particular discussion has sort of started me thinking about it again.
I may review our current logs to see how much time is being wasted.
-- David
/-------\
\ David Bolen
if you're using <2.4.6, you might
try a later version to see if it improves things. Or if you're using
a later version you might try rebuilding with ARENA_SIZE set to 0 to
disable this code to see if your native platform handles it better
somehow.
-- David
/
a flow within rsync at some point. I can't remember just how major
the surgery looked to get the transmission to occur at the point of
computation though.
-- David
/---\
\ David Bolen\
c
to work under Windows at all.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Sta
g.
Beyond that though, I suppose perhaps a linux-oriented group would offer
further suggestions, under the assumption that something must be leading
to the malloc() failing.
-- David
/-------\
\ David Bo
rsync command line option
evolution)
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Stree
a "ulimit -a" show for a server process?
I think under Linux the default settings are in /etc/security/limits.conf,
maybe by default processes on the server are limited to 32MB of memory
or something?
-- David
/----
t; any partial file with data from the original
file on the destination system during an error. No guarantees it
would work as well, since I just took data from the original file past
the size point of the partial copy, but in many cases (growing files)
its a big win. If anyone is interested, I could
Martin Pool [[EMAIL PROTECTED]] writes:
> On 18 Apr 2002, David Bolen <[EMAIL PROTECTED]> wrote:
> > A few caveats - both ends have to support the option - I couldn't make
> > it backwards compatible because both ends exchange information about a
> > sorted fil
e simple enough to just apply manually.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Str
ly intensive to compute, so I think MD4 is more
than sufficient for rsync.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192
eed, I believe).
Thus, it's fairly accurate to compare rsync to performing an ftp or cp
and then doing a full checksum on the file, so one could argue it's
actually more reliable than a straight ftp/cp without the checksum.
-- David
/---
want to update the whole directory, a simple copy may be as
effective as rsync, since you're not benefitting from the selection of
a subset of files.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL P
, but the FAT* variants use local time.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Sta
ssarily the only way to satisfy the GPL, and
if the included cygwin1.dll is simply a binary downloaded from the
Cygwin site, then I think a reference to that site would satisfy
clause 3c of the GPL.
-- David
/---\
\
the existing code.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 0
enerally ends up at the point
where if someone is asking for it, the suggestion is that they implement it
:-)
So I don't think there's any practical workaround with Cygwin.
-- David
/-------\
\ David Bo
It
should take virtually no time for the second (sans the initial
checksum computation and transmission, which to be fair for large
files and small block sizes can be quite significant).
-- David
/---\
\ David Bolen
ck with a modified 2.4.3.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
but if
you do beforehand it's definitely just appended dat that won't be the
case.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
hreads about rsync blocksize.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
if only daily snapshots),
since at least for me rsync would be the only BK project I'd care
about - it's not clear I'd want to bother with the client.
-- David
/-------\
\ David Bolen
the receiver).
- - - - - - - - - - - - - - - - - - - - - - - - -
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
't load :-)
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
x27;m not sure
what a cache would buy.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
by using a larger value
(16K for example).
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford,
t
below 32K).
Not sure how much that would help, although it'll reduce your block
count by about a factor of 2.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx
itable equivalent currently in rsync.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
feeling that trying to reduce the disk space needs is going to be
counter-productive since it goes against the fundamental tradeoffs
inherent in rsync.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTE
t back for possible inclusion
in the main release (after some burn-in time in local use), but there
are some that aren't general purpose enough, so they just stay in our
repository.
-- David
/---\
\ David B
he subject, it's a tad
ambiguous and could certainly include the above scenario.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203)
lta computation.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
ch is
subject to the same timeout setting as the child processes is using
on individual I/O. But the parent won't hear from the child until
it's fully done.
-- David
/-------\
\ David B
ying back database
backups I use a script that creates a lock file before running rsync,
and that lock file is also checked by the backup script.
-- David
/---\
\ David Bolen\ E-mail
K prior to Jet 4.0 and 4K afterwards.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
metry might be affecting timing of information flow, and it would
be interesting if that exacerbated any of the characteristics that his
buffering changes were focused on addressing.
-- David
/---\
\ David Bolen
for the near term :-)
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
something that should probably be handled inside of
rsync better?
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
ng one isn't in your default path but just in the current
directory with some other binary, it might miss it. Doing a flat out
find on your entire system for cygwin1.dll should be a good start
however.
-- David
/----
o access
network shares.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
if some were to occur, perhaps the increased delays could
trigger latent issues in rsync with the pipelining that it uses?
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
|
t;dirties" the file), and then I scale that up a bit based on
database size, to get a reasonable tradeoff between block overhead and
extra transfer upon a change detection.
-- David
/---\
\ David Bolen
copier (in terms of figuring out
what changed), but that's about it.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
ose. (a followup
note corrected the first paragraph as rsync doesn't create any "tree"
structures)
-- David
From: David Bolen <[EMAIL PROTECTED]>
To: 'Lenny Foner' <[EMAIL PROTECTED]>
Cc: [EMAIL PROTECTED]
Subject: RE: The "out of memory" problem wit
;m assuming you would have noticed if you were using that
option previously and stopped doing so :-)
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.
the target
directory at the right date and size. Where --compare-dest could come
into play was if they already existed in the separate comparision
directory, in which case they wouldn't be transferred at all (unless
you were using the -I option).
-- David
/--
fact I believe that other
than for executables (which can swap directly to the source file), all
other memory is automatically backed in the paging system at
allocation time.
-- David
/---\
\ David Bolen
time spent to compute the checksums.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
drive in
a manner that doesn't need the colon.
Alternatively, a low tech approach would just be to switch to the
source drive/directory as your current directory before running the
rsync command.
-- David
/-------\
\
n 2 million files.
True - that's an amazingly large single-directory structure.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
tion that very
first time.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
insane amount of diskspace so I don't care
> anymore)
Might be a really cheap option all things considered nowadays :-)
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
|
backups (updated daily) with the information
in a single file centrally for efficient space usage.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.
output to see it rather
than just trusting the exit code.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
the
process. That way you can schedule it fairly rapidly, but if the
previous run hasn't completed, the next one will be skipped.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECT
re (I earlier thought there was
one) - so please ignore that :-)
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 C
like 28 bytes or so), but the growth algorithm is not
exponential (it just reallocates two entries at a time it looks like).
But I expect compared to other stuff it's in the noise.
-- David
/-------\
\ David Bole
om the receiver. In theory
you could receive them on the sender and place them right into the
hash without affecting the protocol, but that would be more
significant surgery on the source itself.
-- David
/---\
\
n the receiver is changing, but that risk
is already there, just a smaller window. I haven't had a chance to
try the change though yet.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
t file exceeds your available working space.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street,
#x27;s not the way I normally store the files)
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street,
he uncompressed
version. It's interesting that the -9 on the gzip doesn't really help
all that much, although it does decrease the overall file size
slightly.
-- David
/-------\
\ David Bolen\ E
ng this
--rsyncable option on gzip only bumps that up to 1.3x (19MB).
I haven't dug into the patch too deeply yet - perhaps it's designed to
work with a certain block size or something?
-- David
/-------\
\
, this would also imply that the source filesystem would
be the larger of the two.
Of course, I suppose it's also possible that there's something in your
source that rsync isn't syncing up properly - the find comparision
should be able to highlight that.
-- David
/-
it sort of cuts down on the benefit significantly.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.
you can never really get the higher debugging levels locally.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 8
logo :-))
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
r to sender in the first
meta-data transmission.
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
our newly appended data
might match old data already in the file but only at a smaller block
size, using the larger one may prevent that match. So knowing how the
file in question grows and the data it contains may of course qualify
things.
-- David
/-----
significant time issue in my process that unfortunately currently
occurs in series, separated by the time it takes to transmit the
receiver's checksums over an analog line.
-- David
/---\
\ David Bol
looking like it was successful.
-- David
/---\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\ Phone: (203) 708-5192|
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\---/
ilar problems with other tools like CVS
that rename files rapidly after they were just written).
-- David
/-------\
\ David Bolen\ E-mail: [EMAIL PROTECTED] /
| FitLinxx, Inc.\
information (particularly on a
slow link) would be lengthy, it's a more efficient way to guarantee
you only bother sending a file if it is different.
-- David
/---\
\ David Bolen\ E-mail: [EMAI
root if you're
going to use a global exclude.
You can try running with a few -v options (I think -vv may be enough)
to get a glimpse into how rsync is making include/exclude selections.
What happens if you add /foo1/ as a separate line on your include
list?
-- David
/----
79 matches
Mail list logo
