On 16/05/13 03:52, Carl Brewer wrote:
>
> Hello,
> The manual says that rsync treats bind mounts on UNIX (Linux) to the
> same filesystem as being on the same filesystem.
>
> I have a server with a pile of bind mounts to the same filesystem for
> some access control/ease of use for FTP users modify
s algorithm.
Again, the question is meaningless. If you can apply an algorithm one
block at a time, then it's one pass by definition.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
--
Please use reply-all for most replies to avoid omitting the mailing l
the rsync algorithm to both files, and then
make sure the recompression of the target produces the exact same result
would, IMHO, be much more useful than the change you are suggesting.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
--
Please use reply-al
big, and increase the has
size accordingly, thus avoiding the collisions.
In other words - upgrade both sides (but specifically the sender).
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
--
Please use reply-all for most replies to avoid omitting the
issue.
If I understand the scenario you describe correctly, won't running
without -z will merely cause actual undetected data corruption?
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
--
Please use reply-all for most replies to avoid omittin
Jignesh Shah wrote:
Thanks for reply. Could you tell what do you mean by RTFM ctags and
cscope,??
RTFM - Read The Manual
ctags and cscope - utilities whose manual I think you should read.
Creating a new project I think it will have so many errors. We can do
it only if we know the complete co
Jignesh Shah wrote:
Hi Friends,
I have started learning rsync source code but I am finding very
difficult to go back and forth to find the execution flow. I could see
that rsync code is written in UNIX and the compilation is difficult.
Does anybody converted it into Windows Project so that w
Steve Zemlicka wrote:
Thanks Julian and Brad, I will give ntbackup a shot. I've used
rsyncrypto but I'm not a huge fan.
Off topic, but as the author I'd love to hear why.
I don't need the files to be
encrypted except during transit which can be done with just rsync,
right?
Yes. Do rsync o
Matt McCutchen wrote:
On Mon, 2008-10-06 at 18:01 +0200, Shachar Shemesh wrote:
Personally, and this is not something that any shell can solve, I would
love for a way to limit the files that the --server side rsync allows
access to.
It's called an rsync daemon. It can be in
Wayne Davison wrote:
On Sun, Oct 05, 2008 at 06:47:47AM +0200, Shachar Shemesh wrote:
The reason this is brought up is because I'm using rssh
(http://www.pizzashack.org/rssh/) as the user's shell to limit that
user to only be allowed to run rsync.
I looked at the source, and
So, I've done some RTFS, and this is what I've got. I'd still love it if
Wayne could confirm that my understanding of the source is correct.
Shachar Shemesh wrote:
So my questions:
1. Why does rsync need to pass "-e" to the remote side? After all, the
connection is al
$ rsync -e 'ssh -v' lingnu.com:
OpenSSH_5.1p1 Debian-2, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to lingnu.com [199.203.56.105] port 22.
debug1: Connection established.
...
debug1: Sending command: rsync -
Matt McCutchen wrote:
(since rsync does a binary comparison).
rsync as well as the Unix kernel, typically.
I have implemented i18n support in several programs before, I am working
on a draft for BiDi text editing, and I had to look up what
decomposition means. If that's the case, I doubt we
Shane Uys wrote:
Is there a way to automate the rsync password or maybe disable? I am
currently running rsync from a Windows command prompt and would like
to run it from a .bat file. I have read through the config man pages
but not sure if my ssh_config file is even being used. I tried
passw
Aleksey Tsalolikhin wrote:
I've upgraded from rsync 2.6.9 to 3.0.3 on both ends, but memory usage
is still too high.
Why should rsync 3's memory usage depend on the number of files? Does it
keep files it already knows should not be transferred in memory?
If not, then maybe we should hold back
Rob Bosch wrote:
The files are very similar, a maximum of about 5GB of data differences over
80GB. The CPU on both sides is low (3-5 percent) and the memory usage is
low (11MB on the client, not sure on the server).
Full rsync options are:
-ruityz --partial --partial-dir=.rsync-partial --l
Rob Bosch wrote:
I've been trying to figure out why some large files are taking a long time
to rsync (80GB file). With this file, the match process is taking days.
I've added logging to verbose level 4. The output from match.c is at the
point where it is writing out the "potential match at" mes
Hi Wayne, or whoever it is that manages the rsync web site
The "rsync resources" (http://samba.anu.edu.au/rsync/resources.html)
points to a project of mine, rsyncrypto, as a rsync friendly encryption.
Rsyncrypto now has a proper home page, and I would appreciate it if the
link could be updated
Kenneth Simpson wrote:
> Chuck Wolber wrote:
>
>> On Fri, 21 Sep 2007, Kenneth Simpson wrote:
>>
>>
>>
>>> Hi - there's a flag for rsync to compress the files in transit - is it
>>> possible to compress one side (target) with gzip and have rsync still
>>> work correctly?
>>>
>>>
Tobias Oetiker wrote:
> Hi List,
>
> I have updated my rsync fadvise patch which stops rsync from
> ousting all your other data from cache when running large jobs.
>
> I have also written an article about the whole issue.
>
> http://insights.oetiker.ch/linux/fadvise.html
>
> cheers
> tobi
>
>
I
Evan Harris wrote:
> Would it make more sense just to make rsync pick a more sane blocksize
> for very large files? I say that without knowing how rsync selects
> the blocksize, but I'm assuming that if a 65k entry hash table is
> getting overloaded, it must be using something way too small.
rsync
Matt McCutchen wrote:
> Currently, the only way to make rsync do this is with the experimental
> patch "source-filter_dest-filter.diff", which is distributed in
> "patches/" in the rsync source package. If you compile a custom
> version of rsync containing this patch, you can specify bzip2 as the
strange things will happen. Simple moves and
renames inside the directory structure are currently ok, but any
permission change, as well as files being deleted or created, may result
in extremely strange looking files.
> Paul Slootman
>
Shachar
#!/bin/sh
# Run fakeroot with persistent storage of inf
Brad Farrell wrote:
>
> Hi there
>
>
>
> Is there a way with rsync to encrypt data at the source before
> transmitting? Not talking about the actually transmission, but the
> data itself. I’ve got a few department heads that want their data
> secured before it leaves their computer so that no o
l I sent a new
version of rsync was released :-)
Dynamic_hash.diff is available in that one too.
>Also I am planning to install in only the sending machine...and first try
>out.
>
>
Should work.
>Thanks for your feedback.
>lsk.
>
Shachar
--
Shachar Shemesh
Lingn
Julian Pace Ross wrote:
> Thanks everyone for your feedback.
> Seems to me that Alex explained the issue with this perfectly.
I'm afraid that Alex's explanation does not take into account
rsyncrypto's algorithm. If you encrypt two versions of a file, changed
in the first bit of the file between t
khabot wrote:
>Thanks for your response
>I have done this, but I execute
>
>
>>>rsync -avz -e ssh [EMAIL PROTECTED]:/var/mail/ /var/mail
>>>
>>>
>It still askin me for the root password in 10.78.0.117 and for the
>passphrase i, 10.78.0.107
>thank you to help
>
>
Your problem probably has
Matt McCutchen wrote:
>On Tue, 2006-03-28 at 16:58 -0800, Plugger wrote:
>
>
>>We have a server with about 400GB of data that we are trying to backup
>>with rsync. [...] When it runs,
>>however, the load averages on the content1 server continue to grow to
>>the 100s, bringing the server to a pra
[EMAIL PROTECTED] wrote:
>
>
>Thank you for your response.
>
>I compiled rsync 2.6.7 and installed in and that did the trick. I don't
>know if it had the dynamic_hash patch or not.
>
If you did not manually apply it, it did not.
> But I think that I was too
>impatient previously and the 2.6.4 w
Let's try.
[EMAIL PROTECTED] wrote:
>There is one file that is over 45 GB, and it started having
>trouble with that file.
>
> I don't know what the size of the file was before
>it started having trouble
>
>We tried pushing and pulling both, and it gets that error both ways.
>
Rsync has two, un
lsk wrote:
>Also I use the rsync version "rsync version 2.6.5 protocol version 29" does
>this version include this patch "dynamic_hash.diff"" or do we need to
>install it seperately.
>
>
Sorry. You will need to get the 2.6.7 sources, and then apply the patch
yourself and compile rsync.
Please
lsk wrote:
>But I have tried various options including --inplace,--no-whole-file etc.,
>for last few weeks but all the results show me removing the destination
>server oracle datafiles and after that doing an "rsync -vz" from source is
>faster than copying(rsyncing) over the old files that are pre
use tar+gzip that has the "--rsyncable" patch.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, r
Stuart Halliday wrote:
>As long as each machine is set to its own correct default language
>correctly then there isn't a problem I'm aware of.
>
>
But that's exactly what Georgy is complaining about. No amount of
default locale tricks will help you if some of your files are in Spanish
and others
Jamie Lokier wrote:
>While you're there, one little trick I've found that speeds up
>scanning large directory hierarchies is to stat() or open() entries in
>inode-number order. For some filesystems it makes no difference, but
>for others it reduces the average disk seek time as on many common
>fi
Jamie Lokier wrote:
>Hmm. My home directory, on my laptop (a mere 60GB disk), does contain
>millions of files, and it takes about 20 minutes to build the list on
>a good day. 100Mbps network, but it's I/O bound not network bound.
>
>It looks a lot like the number of files is more significant tha
[EMAIL PROTECTED] wrote:
>Hello,
>
> So: each night, from 0:00am to maximum 7:00am, the server will have to
>check the 100Go of files and see what files have been modified, then,
>upload them to the clients. Each file is around 4MB to 40MB in average.
>
>
Are the clients what you call the "mir
John Van Essen wrote:
>Inasmuch as I can follow some of the simpler optimizations, I'm at a
>loss as to what is being so dramatically improved for large files.
>
>Can you write up the little piece that you would add to the NEWS file
>and describe (in layman's terms is possible) what the benefit is
Wayne Davison wrote:
>On Sun, Feb 26, 2006 at 10:34:25AM +0200, Shachar Shemesh wrote:
>
>
>>A line of credit would have been nice :-)
>>
>>
>
>Sure 'nuff.
>
>
Thanks.
>>2. In "sum_buf", sum1 is defined to be unsigned. It see
Wayne Davison wrote:
>http://rsync.samba.org/ftp/unpacked/rsync/patches/dynamic_hash.diff
>
>
A line of credit would have been nice :-)
>One thing this patch does is to (1) leave the array allocated to its
>largest size, (2) use realloc() if we need to make it bigger, (3) make
>the minimum
Wayne Davison wrote:
>Thanks for the patch! Here's some comments:
>
> - You didn't change the size of the "tag" typedef (an unsigned short),
> and your patch makes the value potentially overflow.
>
>
Gotcha. I'm sending an amended patch.
> - For smaller hash-table sizes, your algorithm does
Hi list, and Wayne in particular,
It was almost a year since we had the discussion (with
http://lists.samba.org/archive/rsync/2005-March/011875.html as it's
conclusion) regarding chances for hash collisions and large files. As
now we have someone asking about synching 5TB files, I decided to
actu
Wayne Davison wrote:
>On Thu, Feb 09, 2006 at 03:04:17PM +0100, Paul Slootman wrote:
>
>
>>compare inode and device number. When those are the same, the two files
>>must be hardlinked.
>>
>>
>
>Also, rsync only considers files that have a link count larger than 1
>(see stat()'s st_nlink) sin
Hi all,
I know the question came up once or twice lately, and as I needed
something similar myself, I actually sat down and wrote it. The project
is called "sshpass", and it is available from sourceforge at
http://www.sf.net/projects/sshpass. In a nutshell, it allows
non-interactive use of ssh in
Harish wrote:
>I would like to understand the capabilities of GNU rsync software /
>utility. This is used for syncing file systems / file level data across
>two systems. I specifically would like to know its capabilities in
>syncing files –
>
>
>1) How does it replicate data changes to files –
To whoever it is that maintains the web site.
The page at http://samba.anu.edu.au/rsync/resources.html has a link to
the GNU project management page. The link as a space between the
"http://"; and the host name, which means it cannot be opened.
Shachar
--
To unsubscribe or change op
Wayne Davison wrote:
>On Tue, Nov 01, 2005 at 09:55:06PM -0600, Lawrence D. Dunn wrote:
>
>
>>is it likely, or routine, or will-take-some-time, (or all-of-the-above),
>>for that patch to be vetted and integrated into mainline rsync released
>>code?
>>
>>
>
>I'm currently leaning towards inc
Wayne Davison wrote:
>The patch also makes the new option accepted by the daemon's command-
>line parser, allowing whomever starts the daemon to override the config
>file's "socket option" settings via the command-line.
>
>
Care to elaborate on the security implications? What is the potential
fo
George Georgalis wrote:
>In the archives I see the question about encrypted destination and it's
>mostly answered with the --source-filter / --dest-filter patch by Kyle
>Jones. There are also some proposed updates to the patch.
>
>A lot of these posts 3 years old, is there plans or reasons not to
Joe Pruett wrote:
>>There's something called backuppc (i think backuppc.sourceforge.net)
>>which uses some sort of db backend and has multiple possible transports,
>>rsync is one option. I think it might do what you're looking for.
>>
>>
>
>interesting tool, but it is not what i need. it doe
http://use.perl.org/~Matts/journal/25138
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
Theoretically, turning rsync into a unicode app on Windows could solve
these issues. I doubt it will actually work, however. It is highly
likely to create more problems than it solves, but let's try and find
out what the current problems are before we try to think of a solution.
Hi
I'm assuming that Wayne is the obvious destination for this request. Can
we make the mailing list reject emails from non-subscribers? This would
drastically reduce the amount of spam we receive.
Thanks,
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you back
mercial backup service my
company is running, so you can say I have some experience with it, yes :-).
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.sa
Alun wrote:
Shachar Shemesh <[EMAIL PROTECTED]> said, in message
[EMAIL PROTECTED]:
Reject codes were very common once. Then they were recommended
against. They were recommended against for a reason, that reason
being that they expose the user base to password and other guessing.
#x27;t like something defined by the RFC is
counter-productive and does more to hurt spam fighting than to help it.
Now, this is getting off topic for rsync, so please do feel free to send
me your reply privately.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you bac
the very
essential NACK SMTP has from all servers, as per spamcop's request.
-John
[EMAIL PROTECTED]
Personal Opinion Only
Same here.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsu
at file with the "--block-size=524288" (i.e. - a 0.5MB
block size), and please report whether rsync's behavior had improved,
and in particular, how does it rate against vanilla ftp.
Thanks,
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backe
hile your largest file has a load of 53%. You
shouldn't see any problems when using rsync (at least, not the type of
problem I'm talking about).
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingn
you run across this problem, please post on list, as we need someone
to experience this problem in order to try and fix it.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or c
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
sorry about the noise.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb
or follows that I have to amend my previously proposed hash
table size choosing formula. The new formula is:
(numblocks/8+1)*10+1
And you're done. Of course, this can also be written as:
(numblocks/8)*10+11
Which is slightly more efficient.
Shachar
--
Shachar Shemesh
Lingnu Open S
ch so it can be
evaluated (hoping I'll manage to find my way around, that is).
..wayne..
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mail
Wayne Davison wrote:
On Sat, Mar 05, 2005 at 02:18:01PM +0200, Shachar Shemesh wrote:
However, if you choose to start with the 32 bit rolling hash and mod
that, you will have problems. The rolling checksum has two distinct
parts, and modding will only pull info from the low order bits
Wayne Davison wrote:
On Thu, Mar 03, 2005 at 10:18:01AM +0200, Shachar Shemesh wrote:
And I'm suggesting making it static, by adjusting the hash table's
size according to the number of blocks.
The block-size?
Definitely not! I was talking about the hash table load. I.e. -
er doing mass-lookups on the second.
Am I missing something basic here?
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
B
Well, me either, as the rsync job processes both this gigantic file
and other smaller ones.
If you don't specify block sizes, this should not be a problem.
Whoa, it that the subject? I thought the subject was solving my
problem
Not for four or five messages, no :-)
Shachar
--
Trying to solve his use of possibly
non-optimal values won't help rsync, though it will help him. Let's keep
this part of this thread on it's subject - is the hash table size
optimal? I don't see how modifying the block sizes is going to change
this question.
320 buckets, it should go into bucket 11. Assuming the checksums
are sufficiently random-like, this algorithm is good enough.
Cheers,
Kevin
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe
#x27;t even have to change the line protocol in any way.
Am I misreading the code?
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/
Shachar Shemesh wrote:
No, because the rsync algorithm can detect single byte moves of this
700 bytes block.
I will just mention that I opened the ultimate documentation for rsync
(the source), and it says that the default block size is the rounded
square root of the file's size. This
question is "why".
How much memory is on the machines? Try to bring the block size up to
1MB. This will mean you will have only 524 thousand blocks, which may
prove more manageable.
Best regards,
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed
eiver, and all other blocks will have a one byte
offset (which rsync will detect, and save the traffic). In short, we see
that the 700 number has almost nothing to do with the application that
the file belongs to.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you
y bit of trivia for you. Ssh's key authentication will not work if
either the "authorized_keys" file, or the .ssh directory it's in, are
world readable or executable. I also think they have to be owned by the
user doing the authentication.
Shachar
--
Shachar Shemes
the file system. It doesn't care what the file system is.
If it's a samba mount, then that's ok. I am not aware of a FTP
filesystem, and therefor can't say that it will work over ftp.
Any help to any questions will be appreciated! Thank You!
Regards,
Fred
Shach
something the program itself prints, that they have these rights.
I hope this answers your question.
Also, don't even think about relying on this small email to base your
business on this answer. If in doubt, get a lawyer.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consultin
coding to the --rsyncable patch, but it should be
relatively simple to do.
That's all I have for now - I'm running tests this evening to independently
adjust the window and block mask size, and to test a different data file just
to make sure I'm not way off-base (i.e. make sure t
with a
1 second resolution --modify-window=1 is useful.
Are you on FAT?
Thanks, Tuc
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html
--
To unsubscribe or change options: https://lists.samb
ect's summary page at
http://sourceforge.net/projects/rsyncrypto.
Thanks, and sorry about the noise,
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before
Martin Schröder wrote:
On 2005-02-04 11:51:20 +0200, Shachar Shemesh wrote:
What distro is this? If it's Debian, gzip has an option called
"--rsyncable". This makes changes to the uncompressed file local in the
This is a debian-only patch which doesn't change the gz
irs... | gzip --rsyncable >
file.tgz"
Enjoy
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
e actual
tree scanning and copying, but it will soon. It goes without saying that
any help would be appreciated.
--
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, r
83 matches
Mail list logo
