Randy,
A couple of things:
1) rsync will do block checksums replacements of files, but in the case
of an
.iso, please consider that if one file grows to use an extra block, it
will
offset the blocks of all the files after it on the CD (N representing
the new
block):
AAABBBBBBBCCCCCCDDDDEEEEE...
AAABBBBBBBNCCCCCCDDDDEEEEE...
This would cause all of the blocks including N and following N to be
resent,
because they do not match. This, of course, is assuming that there are
no new
files, no files removed, and all the files are in exactly the same
order, as
these conditions would cause offset problems with the rsync block
checksum
algorithm as well.
2) I think it was mentioned that there was a bug with the implementation
of
the timeout option that would cause problems when waiting for a checksum
of a
large file. Maybe that was fixed...
The only way I think this would work would be to:
1) nfsmount the source .iso's directory
2) ISO9660 mount the source .iso in the nfsmount
3) mount the old beta .iso in another location
4) rsync from the contents of the source iso to the old beta copy you
have
This way, you are comparing the individual files, and the location of
the
files on the .iso does not matter. Of course, nfsmounting from a mirror
site
is usually not possible, so this method isn't very useful.
Better to use ftp.
John E. Mayorga
Randy Kramer wrote:
> I had asked questions on the expert, newbie, and rsync list about using
> rsync to correct a bad md5sum on an iso image. I finally got it to work
> (instructions below). Somebody told me (in an email) that it will not
> work to upgrade from one version of an iso to another (like Mandrake 8.0
> beta 2 to final), but I don't believe that -- I believe it will work but
> I haven't tried it yet -- thought I'd wait a few days or longer to let
> traffic on the mirrors die down.
>
> When I try it, I will:
>
> -Make an archive copy of my existing local copy of the iso. Put a
> working copy in a directory with space for at least one additional copy
> of the iso and rename it to match the name of the iso that I am trying
> to "download".
>
> -If I have space for two copies, I will put one in a subdirectory of
> this directory and then specify the subdirectory name in the
> --compare-dest option of rsyc.
>
> (See my notes below about the --partial and the --compare-dest options
> -- if you specify the partial option (which I think is a good idea) but
> your transfer is interrupted, the original iso will be replaced by the
> partially transferred copy. Sometimes it will make sense to continuse
> rsync with the partially transferred copy (if the transfer was almost
> completed), but more often it will make sense to restart rsync from the
> beginning, using the original iso. If you don't save a copy somewhere,
> you will be forced to continue with the partially transferred iso. When
> you continue with a partially transferred copy, you are no longer
> reaping the benefit of rsync but basically doing a "pure download" of
> the remainder of the file. If the --compare-dest option worked as I
> thought it might, this would not be an issue, but, so far, the
> --compare-dest option has not worked for me (more discussion below).)
>
> -Look at the mirror site to find the full name and path of the iso I
> want to download (use a web browser).
>
> -Find the first directory in the rsync path by running:
>
> rsync carroll.cac.psu.edu::
>
> (The rsync server establishes a relative path to the iso file -- in the
> rsync command you must specify this relative path, not the full path.)
>
> -Use a command line like (with the PWD set to the directory I want to
> download to):
>
> rsync -a -vv --progress --partial [--compare-dest=<subdirectory_name>] \
> carroll.cac.psu.edu::mandrake-iso/mandrakefreq-i586-20010316.iso \
> mandrakefreq-i586-20010316.iso
>
> (Note the full path to the file (on carroll) is something like
> /pub/.../.../mandrake-iso/mandrakefreq-... -- the rsync path (on
> carroll) starts with mandrake-iso.)
>
> (Note the double colon. That works on carroll. There are other
> variations for specifying the rsync server, some with a single colon,
> some using the word rsync. You don't get a choice, you have to specify
> the one that is right for the server you want to download from and the
> communication method you want to use -- ssh, rsh, or? See the man pages
> for rsync -- when I learn more about these other options I will modify
> these instructions. Note that I don't think anybody needs to use ssh to
> download a Mandrake iso -- who are you keeping a secret from?)
>
> (The --compare-dest parameter is optional, see discussion above and
> below.)
>
> You must make sure that the timestamp (or filesize?) on your local copy
> of the file differs from the remote copy, otherwise rsync will think the
> local and remote iso are the same and not attempt to sync them. Touch
> the local file if necessary.
>
> -Do not specify the -c option!
>
> (You might think that you can use the -c option to force a full checksum
> check on the files. This did not work for me. The problem for me was
> that I got a message "unexpected EOF on read-timeout", and a short time
> later the rsync process quit. What I didn't realize until later is that
> this error message means the rsync server has quit. The rsync server
> often quit for me when I used the -c option, generally because something
> on the server end decided to kill rsync. I don't know whether this was
> due to some parameters of the rsync server, or something like an
> external watchdog watching the rsync server and deciding it was not
> making sufficient progress, but I do know that if you specify the -c
> option it uses a lot of processor resources on the server side, and, at
> least for me and carroll, I often got the "-timeout" message until I
> stopped specifying the -c.)
>
> -The -a option stands for archive and is a way of specifying several
> options at once. It will make sure that (IIRC) the file size,
> timestamp, owner, group, permissions, and ?? are corrected by rsync.
> (Note that if you try to use rsync in dos/Windows you may run into a
> problem because dos/Windows can only resolve timestamps to an even
> number of seconds -- if the original iso file has a timstamp with an odd
> number of seconds, you will have trouble getting the md5sum to come out
> correctly. (There may be some workarounds -- I used rsync under Linux
> so I did not have to deal with any issues here.)
>
> -I read some of Andrew Tridge's thesis (somewhere at rsync.samba.org).
> Rsync divides a file into blocks (IIRC, somewhere between 3000 and 8000
> bytes per block (and there is a command line parameter to adjust the
> block size). As long as there are blocks in this size range that are
> identical between the remote file and the local file, rsync will find
> those duplicate blocks and not transfer them, thus saving the amount of
> time that would otherwise be required to transfer those blocks. I
> suspect that there are enough matching blocks between beta 3 and the
> final Mandrake 8.0 to make rsync worthwhile. Some other notes about
> using rsync:
>
> --If you are trying to rsync a Linux text file with a dos/Windows text
> file, rsync may be useless if one copy uses the <cr><lf> line ending
> convention (dos) and the other file uses the <lf> line ending convention
> (Linux). (If lines are 80 characters, no 3000 character blocks will
> match. There are some potential workarounds to this problem that I will
> not discuss here, except to suggest using sed on your local copy to
> convert all <cr><lf>s to <lf>s or vice versa, if necessary, before
> running rsync. I would not expect any problems like this while dealing
> with an iso, and I did not have any such problems even though I
> downloaded the file using rsync under Linux, and then copied the file to
> a Windows machine to burn the CD. (When I got the md5sum right in
> Linux, it stayed correct after the transfer to Windows.))
>
> --rsync must have space for at least one extra copy of the iso file in
> your local directory. As rsync works, it copies portions of the file
> from the server or from the local copy (for matching blocks) and
> constructs a new copy of the iso under a temporary file name. When
> rsync has reconstructed the file, it deletes the original file, and
> renames the reconstructed copy to match the original file's name, date,
> etc.
>
> If rsync is interrupted before completion, the temporary file is lost,
> and rsync starts over from scratch unless you have specifed the
> --partial option. I recommend you specify the --partial option, but if
> the transfer is interrupted you will have to make a decision. First see
> my notes (below) about the --compare-dest option. Assuming the
> --compare-dest option does not work the way I hoped it would work, you
> must choose whether to start the rsync process over from the beginning
> (by replacing the partially transferred file with the original local
> copy of the iso), or continuing from where you left off. If you
> continue from where you left off, the remainder of the transfer will be
> basically a "pure download" -- rsync will have no local copy of the file
> to check for block matches.
>
> In reading about the --compare-dest option, it sounded like I could keep
> two local copies of the file I wanted to update, one in the download
> directory and one in a subdirectory specified in the --compare-dest
> option. I hoped, that if I restarted rsync after an interruption, it
> would continue building the partially transferred file but use the file
> in the --compare-dest directory for comparison to the original and as
> "raw material" to avoid transferring duplicate blocks. It did not work
> for me, but it might have been because:
>
> -I did not set things up properly -- my hard disk does not have space
> for three copies of the iso, so I burned a CD-Rom and used it for the
> --compare-dest copy. If I had more hard disk space, I would try this
> again with the third copy in a subdirectory below the directory I am
> transferring the iso into.
>
> -The --compare-dest option only works for entire files -- if you are
> transferring multiple files, those that have been successfully
> transferred are recognized as completed by rsync and are not rechecked,
> those that have not been transferred use the files in --compare-dest as
> the raw material for the resumption of the rsync process.
>
> -I misunderstood the description and purpose of --compare-dest.
>
> -There is a bug in --compare-dest.
>
> Hope this helps!
>
> After I get my TWiki up and working, I will rewrite these instructions
> and put them on my TWiki site. (I apologize because these are not as
> clear as I would like -- they need to be rewritten -- I just don't have
> the time right now.) I would appreciate corrections or clarifications
> to these instructions, especially with respect to the --compare-dest
> option.
>
> Randy Kramer
>
> CB wrote:
> >
> > On Thu, Apr 19, 2001 at 02:27:37PM -0700, Rusty Carruth wrote:
> > > There was a question a while back on the rsync list (I think)
> > > about updating iso images using rsync, thus keeping you from
> > > having to load an entire 600meg iso image when you already
> > > have the beta just before that one.
> >
> > I don't think rsync works that way. It compares on a file by file basis
> > and when it finds a file that's different, it downloads it and replaces
> > it on the local machine. Since the iso file changes, it downloads the
> > new iso file. The whole 600 megs. You're not getting the savings you
> > think you are.
> >
> > The one advantage you might get is that ftp might be traffic
> > shaped, whereas rsync might not. Notice the two _might_s in there.
> > Those were strictly guesses.
> > --
> > Blue skies... Todd
> > | Get a bigger hammer! | Sometimes you get what you want. |
> > | http://www.mrball.net | Sometimes you get experience. |
> > | http://faq.mrball.net | --unknown origin |
--
John E. Mayorga
HSA Email Administrator
[EMAIL PROTECTED]
[EMAIL PROTECTED]
+1.408.922.4031
