Re: Using the rsync checksums for handling large logfiles.
Kurt Roeckx ([EMAIL PROTECTED]) said, in message <[EMAIL PROTECTED]>: > > Ever considered using syslog, and sending all data to the central > log server? Thanks for the suggestion, but standard syslog is UDP and so not reliable (I know about syslog-tng and friends) and some of the logs that we want aren't generated via syslog anyway. Cheers, Alun. -- Alun Jones [EMAIL PROTECTED] Systems Support, (01970) 62 2494 Information Services, University of Wales, Aberystwyth pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using the rsync checksums for handling large logfiles.
On Fri, Nov 14, 2003 at 02:30:32PM +, Alun wrote: > > Dear all, > > I've only just joined this list, but I can't find any mention of this > idea anywhere else, so I thought I'd just post here before getting too > deep into programming and possibly reinventing the wheel. > > Here at Aber, we have around 30 unix and linux servers doing core services. > Each one is maintaining its own logfiles and, for various reasons, we want to > keep these on the servers' local disks, with each server having its own log > rotation policy. We also have a box which acts as a central repository of > logfiles. A script runs every 10 minutes on this box and pulls whatever's > been appended to the logfile during the previous 10 minutes across to the > log server. At the moment, all this is based on a database which remembers > the old size of the remote logfile and just requests whatever's in the file > after that point. Things get a bit complicated when you try to allow for the > remote log rotation and, if anything goes wrong, the database can get out of > sync with reality, leading to lost log data. > > Now, as I see it, rsync would be a useful tool for this job, except that it > would mean that we end up scanning a large file on both servers every 10 > minutes. I've played around on paper and come up with a reversed version of > the rolling checksum (i.e. given a_k, b_k calculate a_k-1 and b_k-1). The > idea is that I calculate the checksum of the final block of the log on the > loghost and pass that to the remote host. A program there starts reading > backwards through the logfile, until it finds a checksum match. It then > feeds back everything in the file after that point. If rotation has happened > on the remote host, the client could start reading backwards through the > rotated log etc. > > I think this would give me a lot of advantages over our current system (it > would be very robust, if nothing else). It also seems like something that > could be added to rsync itself fairly easily. Some sort of flag to say "I > guarantee that the only differences in the remote file are at its end" > could save huge amounts of disk I/O when used for log synchronisation. Does > anyone know of anything similar, or of any plans to make rsync capable of > reading backwards in this manner? If not, I'll go my own way with my hacked > checksum. The rsync protocol wouldn't lend itself to this. Overall, rsync is going to be less efficient than a utility that simply sends what needs to be appended to reach the same length. I'd have the receiver send the file list with lengths and the sender would compare with it's copy and if the file is shorter send the whole file with a flag to say "rotated", but if the sender's file is longer it would open, seek to the receivers length and then send the rest of the file. That would be much more efficient than rsync with the block checksums, file scanning and copying to tempfiles that get renamed. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using the rsync checksums for handling large logfiles.
jw schultz ([EMAIL PROTECTED]) said, in message <[EMAIL PROTECTED]>: > > The rsync protocol wouldn't lend itself to this. Overall, > rsync is going to be less efficient than a utility that > simply sends what needs to be appended to reach the same > length. > > I'd have the receiver send the file list with lengths and > the sender would compare with it's copy and if the file is > shorter send the whole file with a flag to say "rotated", > but if the sender's file is longer it would open, seek to > the receivers length and then send the rest of the file. > That would be much more efficient than rsync with the block > checksums, file scanning and copying to tempfiles that get > renamed. Dear all, Thanks for your thoughts. There are problems with relying purely on lengths (and these are what got me thinking about the checksums) - if you rotate and then a service gets hit hard before the next sync, it's possible that the new file is actually longer than the old was prior to rotation. This has happened to us a couple of times where some problem or other has caused the updates to not happen as often as they should. If, instead, the server passes a block checksum (of the last thing it has) to the client and the client then works backwards through everything it has, looking for a match, then a temporary failure can't cause data loss until a log has actually rotated out of existence. It sounds like I'm not missing some subtlety in the use of rsync, so I think I'll hack on at implementing my backwards search idea. First attempts, mmapping the file in 32Mbyte chunks and using the reversed checksum, suggest I can scan backwards at around 13Mbytes/second on a 1GHz Pentium. Cheers, Alun. -- Alun Jones [EMAIL PROTECTED] Systems Support, (01970) 62 2494 Information Services, University of Wales, Aberystwyth pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --daemon and logfile that can't be created
If the rsyncd.conf has a line such as:
log file = /var/log/rsync/log
and /var/log/rsync doesn't exist or isn't a directory (or the log file
can't be opened for any other reason), then there's no warning
whatsoever, as rsync forks itself into the background before checking
the config, opening the log file, etc.
Worse still, it gets a SIGSEGV, and dumps core. Here's a strace from a
fairly recent CVS:
15154 umask(022)= 0
15154 open("/var/log/rsync/log", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0666) = -1
ENOTDIR (Not a directory)
15154 umask(0) = 022
15154 umask(022)= 0
15154 open("/var/log/rsync/log", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0666) = -1
ENOTDIR (Not a directory)
15154 umask(0) = 022
15154 getpid() = 15154
15154 time(NULL)= 1069070906
15154 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
15154 +++ killed by SIGSEGV +++
15153 --- SIGCHLD (Child exited) @ 0 (0) ---
15153 waitpid(-1, [WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV && WCOREDUMP(s)], WNOHANG)
= 15154
15153 waitpid(-1, 0xb864, WNOHANG) = -1 ECHILD (No child processes)
15153 sigreturn() = ? (mask now [])
15153 exit_group(0) = ?
1) It's a bother that it first daemonizes itself and then checks the
config etc., giving no feedback whatsoever that it's not running.
2) core dumps are never good :-(
Here'a a proposed patch to fallback to syslog if opening the log file
fails:
--- ../cvstree/rsync/log.c 2003-03-25 03:28:54.0 +0100
+++ ./log.c 2003-11-17 13:31:15.0 +0100
@@ -134,9 +134,9 @@
static void logit(int priority, char *buf)
{
- if (logfname) {
- if (!logfile)
- log_open();
+ if (logfname && !logfile)
+log_open();
+if (logfile) {
fprintf(logfile,"%s [%d] %s",
timestring(time(NULL)), (int)getpid(), buf);
fflush(logfile);
@@ -164,9 +164,11 @@
if (logfname) {
if (*logfname) {
log_open();
- return;
+ if (logfile)
+return;
+/* fall back to syslog if opening log file fails */
}
- logfname = NULL;
+logfname = NULL;
}
#ifdef LOG_NDELAY
@@ -191,6 +193,12 @@
int old_umask = umask(022 | orig_umask);
logfile = fopen(logfname, "a");
umask(old_umask);
+if (!logfile) {
+char buf[512];
+sprintf(buf, "logfile %.400s could not be opened, errno %d",
logfname, errno);
+logfname = NULL;
+logit(LOG_ERR, buf);
+}
}
}
Paul Slootman
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.6 hanging
I get the same exact results on redhat 7.3 and redhat 8.0 so I dont think it is a redhat issue. Also if I use an earlier version of rsync (version 2.3.4), then I still get the same problem, but it doesn't crash as quickly as the 2.5.6 version. --- Hardy Merrill <[EMAIL PROTECTED]> wrote: > FYI - RHL 6.1 is no longer supported - from a Red > Hat > perspective that is :-( See the Red Hat "Errata: > Security > Alerts, Bugfixes, and Enhancements" page here > >http://www.redhat.com/apps/support/errata/ > > You may indeed get help here since this list is an > rsync > list. If I knew what your problem might be I'd try > to > help you myself, but I don't so hopefully someone > else will > chime in with some ideas for you to try. > > One question I do have is what version of rsync is > running on the ftp server? Also, if you know the > exact > file it gets stuck on, how big is that file? > > -- > Hardy Merrill > Red Hat, Inc. > > Zachary Denison [EMAIL PROTECTED] wrote: > > I am using rsync 2.5.6 on redhat linux 6.1 with > 1gb > > ram. What I would like to do is mirror a > directory on > > the server to another server for backup purposes. > The > > server is an ftp server and the main directory in > > question is 77GB consisting of 391000 > subdirectories > > and files. when I run rsync with the following > > command: > > > > /usr/bin/rsync -av --rsh=ssh --stats --progress > > --rsync-path=/usr/bin/rsync --delete --force > > --exclude=logs/ /ftpfs/ 10.10.1.237:/ftpfs/ > > > > it starts to transmit some files and then gets > stuck. > > Actually now it is stuck on one file and every > time I > > run it it it "freezes" as it is transmitting the > file: > > as in the example below: > > > > 196608 22% 629.92kB/s0:00:01 > > > > then it just hangs.. > > > > I have 1gb of ram in the machine so it shouldn't > be a > > lack of memory issue. I would greatly appreciate > any > > ideas. thank you.. > > > > Zach. __ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.6 hanging
Zachary Denison [EMAIL PROTECTED] wrote:
>
> I get the same exact results on redhat 7.3 and redhat
> 8.0 so I dont think it is a redhat issue. Also if I
> use an earlier version of rsync (version 2.3.4), then
> I still get the same problem, but it doesn't crash as
> quickly as the 2.5.6 version.
You didn't answer my questions - assuming you are running
rsync 2.5.6 on the "source" machine:
1. what version of rsync is running on the ftp server?
("destination" machine)
2. if you know the exact file that it gets stuck on,
then how big is that file?
Again, not sure I can help you yet, but the answers to
those questions may give either me or others the clues
needed to help you.
--
Hardy Merrill
Red Hat, Inc.
>
>
>
> --- Hardy Merrill <[EMAIL PROTECTED]> wrote:
> > FYI - RHL 6.1 is no longer supported - from a Red
> > Hat
> > perspective that is :-( See the Red Hat "Errata:
> > Security
> > Alerts, Bugfixes, and Enhancements" page here
> >
> >http://www.redhat.com/apps/support/errata/
> >
> > You may indeed get help here since this list is an
> > rsync
> > list. If I knew what your problem might be I'd try
> > to
> > help you myself, but I don't so hopefully someone
> > else will
> > chime in with some ideas for you to try.
> >
> > One question I do have is what version of rsync is
> > running on the ftp server? Also, if you know the
> > exact
> > file it gets stuck on, how big is that file?
> >
> > --
> > Hardy Merrill
> > Red Hat, Inc.
> >
> > Zachary Denison [EMAIL PROTECTED] wrote:
> > > I am using rsync 2.5.6 on redhat linux 6.1 with
> > 1gb
> > > ram. What I would like to do is mirror a
> > directory on
> > > the server to another server for backup purposes.
> > The
> > > server is an ftp server and the main directory in
> > > question is 77GB consisting of 391000
> > subdirectories
> > > and files. when I run rsync with the following
> > > command:
> > >
> > > /usr/bin/rsync -av --rsh=ssh --stats --progress
> > > --rsync-path=/usr/bin/rsync --delete --force
> > > --exclude=logs/ /ftpfs/ 10.10.1.237:/ftpfs/
> > >
> > > it starts to transmit some files and then gets
> > stuck.
> > > Actually now it is stuck on one file and every
> > time I
> > > run it it it "freezes" as it is transmitting the
> > file:
> > > as in the example below:
> > >
> > > 196608 22% 629.92kB/s0:00:01
> > >
> > > then it just hangs..
> > >
> > > I have 1gb of ram in the machine so it shouldn't
> > be a
> > > lack of memory issue. I would greatly appreciate
> > any
> > > ideas. thank you..
> > >
> > > Zach.
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.6 hanging
Hi Hardy,
The version I am running on the destination machine is
also rsync 2.5.6. The destination machine has 4GB ram
in it and is running redhat 8.0. Also it gets stuck
on all different types of files, small and large.
sometimes the filesize is 200k and sometimes its
several megabytes.
I also tried different configurations, e.g. using rsh
instead of ssh, using no compression instead of
compression. But to no avail.
Any hints would be greatly appreciated. Thanks
Zach.
--- Hardy Merrill <[EMAIL PROTECTED]> wrote:
> Zachary Denison [EMAIL PROTECTED] wrote:
> >
> > I get the same exact results on redhat 7.3 and
> redhat
> > 8.0 so I dont think it is a redhat issue. Also if
> I
> > use an earlier version of rsync (version 2.3.4),
> then
> > I still get the same problem, but it doesn't crash
> as
> > quickly as the 2.5.6 version.
>
> You didn't answer my questions - assuming you are
> running
> rsync 2.5.6 on the "source" machine:
>
>1. what version of rsync is running on the ftp
> server?
> ("destination" machine)
>
>2. if you know the exact file that it gets stuck
> on,
> then how big is that file?
>
> Again, not sure I can help you yet, but the answers
> to
> those questions may give either me or others the
> clues
> needed to help you.
>
> --
> Hardy Merrill
> Red Hat, Inc.
>
>
> >
> >
> >
> > --- Hardy Merrill <[EMAIL PROTECTED]> wrote:
> > > FYI - RHL 6.1 is no longer supported - from a
> Red
> > > Hat
> > > perspective that is :-( See the Red Hat
> "Errata:
> > > Security
> > > Alerts, Bugfixes, and Enhancements" page here
> > >
> > >http://www.redhat.com/apps/support/errata/
> > >
> > > You may indeed get help here since this list is
> an
> > > rsync
> > > list. If I knew what your problem might be I'd
> try
> > > to
> > > help you myself, but I don't so hopefully
> someone
> > > else will
> > > chime in with some ideas for you to try.
> > >
> > > One question I do have is what version of rsync
> is
> > > running on the ftp server? Also, if you know
> the
> > > exact
> > > file it gets stuck on, how big is that file?
> > >
> > > --
> > > Hardy Merrill
> > > Red Hat, Inc.
> > >
> > > Zachary Denison [EMAIL PROTECTED]
> wrote:
> > > > I am using rsync 2.5.6 on redhat linux 6.1
> with
> > > 1gb
> > > > ram. What I would like to do is mirror a
> > > directory on
> > > > the server to another server for backup
> purposes.
> > > The
> > > > server is an ftp server and the main directory
> in
> > > > question is 77GB consisting of 391000
> > > subdirectories
> > > > and files. when I run rsync with the
> following
> > > > command:
> > > >
> > > > /usr/bin/rsync -av --rsh=ssh --stats
> --progress
> > > > --rsync-path=/usr/bin/rsync --delete --force
> > > > --exclude=logs/ /ftpfs/ 10.10.1.237:/ftpfs/
> > > >
> > > > it starts to transmit some files and then gets
> > > stuck.
> > > > Actually now it is stuck on one file and every
> > > time I
> > > > run it it it "freezes" as it is transmitting
> the
> > > file:
> > > > as in the example below:
> > > >
> > > > 196608 22% 629.92kB/s0:00:01
> > > >
> > > > then it just hangs..
> > > >
> > > > I have 1gb of ram in the machine so it
> shouldn't
> > > be a
> > > > lack of memory issue. I would greatly
> appreciate
> > > any
> > > > ideas. thank you..
> > > >
> > > > Zach.
__
Do you Yahoo!?
Protect your identity with Yahoo! Mail AddressGuard
http://antispam.yahoo.com/whatsnewfree
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.6 hanging
Hi Zach, On Mon, Nov 17, 2003 at 09:24:12AM -0800, Zachary Denison wrote: > The version I am running on the destination machine is > also rsync 2.5.6. The destination machine has 4GB ram > in it and is running redhat 8.0. Also it gets stuck > on all different types of files, small and large. > sometimes the filesize is 200k and sometimes its > several megabytes. Have you tried the CVS version? My impression is that many bugs have been patched. If you can install the CVS code on both source and destination, that may be worth a try. -Andy -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync 2.5.6 hanging in cygwin on Windows shares
rsync 2.5.6 is hanging in cygwin, at the end when it should print a summary of bytes sent etc and exit. It has successfully copied all the files before hanging. It does this every time when copying to or from a Windows share. (It sometimes hangs when copying from one local drive location to another, but I have not tested this as thoroughly.) The Windows shares are almost certainly on Windows machines, probably win 2000, but are not within my control (managed by an out-sourced IT dept). I am running cygwin on a Windows 2000 PC. I have worked around this by reverting to rsync 2.5.5 that has always worked perfectly for me. The suggested rsync.exe replacement at http://sourceforge.net/projects/backuppc/ also hangs. (If you need more details, eg exact rsync command I am using, please ask.) Apologies if this problem is already known about; I searched the archives but found nothing that quite matches what I have observed. -- John Hunt Stokesley, N Yorks, UK e-mail: [EMAIL PROTECTED] URL: http://www.yoredale.uklinux.net/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
selling domain..
Are you perhaps interested in buying SAMPLESTORES.COM ? I'm selling the domain through cenion for $89.95. If you are interested signup here: http://www.cenion.com/transfer/2/e/?SAMPLESTORES.COM Jamie -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --include=PATTERN not working!
but I do have the recursive option on -rvaz > On Fri, Nov 14, 2003 at 10:02:56PM -, Haisam K. Ido wrote: >> Why does'nt this work? >> >> rsync -rvaz --include="*.lis" --exclude='*' [EMAIL PROTECTED]:/path/ . >> receiving file list ... done >> client: nothing to do: perhaps you need to specify some filenames or the >> --recursive option? > > Looks like it is working perfectly. You ask it to only > rsync file matching *.lis only if they are directly in the > directory specified but not in any subdirs. There aren't > any so there is nothing to do. > > > -- > > J.W. SchultzPegasystems Technologies > email address: [EMAIL PROTECTED] > > Remember Cernan and Schmitt > -- > To unsubscribe or change options: > http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Win2k Scheduler not able to execute rsync over SSH
Hi there, I have tried enough of it and couldnt get any where. I have a Redhat 9.x box with rsync I want to backup contents on the RH to Win2K box. I have copied required cygwin DLL's and rsync.exe,ssh-keygen.exe and ssh.exe. I set up my pub rsa id and all that stuff. My problem is here: I have a batch file which logs itself. The batch file contains some logging commands and rsync -avze ssh [EMAIL PROTECTED]:/directory/. /local_dir This command is executed fine at the command prompt. (Doesnt ask for password or passphrase as it is all setup.) when I execute this batch file thorugh the Win2K Task scheduler all the other commands in the batch file are executed except the above command. Dont know why. I have another job which backs up from anther server and contains rsync -avz server::share_name/directory/. /local_dir And this executed fine from command prompt and task sheduler. Only if I use ssh as transport the command is not executing. TIA for any help and advice. Raja -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: selling domain..
Jamie Bolden wrote: Are you perhaps interested in buying SAMPLESTORES.COM ? I'm selling the domain through cenion for $89.95. If you are interested signup here: http://www.cenion.com/transfer/2/e/?SAMPLESTORES.COM Jamie I get enough of this SPAM daily from other sources. Can someone please remove this annoying little man? Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
