On Sat, Sep 23, 2006 at 12:27:21PM +0200, frantisek holop wrote:
> hi there,
>
> i have just moved a couple of big files using scp from my server
> to my notebook. i left it going all night and when i came back
> i had a no space left on device. so i made some more space
> and before restarting the transfer i fired up man scp because
> for some reason i couldn't remember the parameter for resuming
> copying. apparently, because there isn't...
>
> so i started to experiment a bit, start the download, abort it,
> start it again, abort. and i still can't tell what's happening.
>
> amaaq> scp server:bigfile.iso .
> bigfile.iso 0% 432KB 100.1KB/s 1:59:10
> ETA^CKilled by signal 2.
> amaaq> ls -la bigfile.iso
> -rw-r--r-- 1 f wheel 606208 Sep 24 11:19 bigfile.iso
>
> amaaq> scp server:bigfile.iso .
> bigfile.iso 0% 1824KB 112.3KB/s 1:46:04
> ETA^CKilled by signal 2.
> amaaq> ls -la bigfile.iso
> -rw-r--r-- 1 f wheel 1884160 Sep 24 11:20 bigfile.iso
>
> amaaq> scp server:bigfile.iso .
> bigfile.iso 0% 416KB 73.8KB/s 2:41:48
> ETA^CKilled by signal 2.
> amaaq> ls -la bigfile.iso
> -rw-r--r-- 1 f wheel 1884160 Sep 24 11:20 bigfile.iso
>
>
> when the file is 1884160 bytes, and i restart the copying,
> the file is not truncated but there is no seek to its end either,
> so what is being downloaded and where is it put until it catches
> up with the current position? is it just discarded?
I presume scp just starts writing at the beginning of the file, and
keeps at it until the whole file is transferred. Of course, it does
truncate the file if the desired length is smaller than the length
before the copy starts.
> i think it would be nice to document this ambiguity in the scp
> man page...
Not really necessary, IMHO. What happens when the copy fails is
currently undefined; and I don't see a problem with that.
> this unfinished download reminded me of another issue.
> some of the programs which do downloads started using the
> *.part or *.filepart filenames until the download is finished.
>
> i realize that for some, this might seem as an unnecessary
> wasting of resources (think moving lots of small files) but
> i think it's good to know if the transport was really finished
> and the files really are the ones i started copying and not
> only its parts.
The scp interface is quite clear about that.
> was there any consideration to give scp similar functionality?
>
>
>
> if none of these features (resuming, renaming) do make sense
> for scp, what do other people use for transporting (esp. big)
> files?
I use sftp (and occasionally scp or tar-over-ssh), ftp, Subversion
(svn), unison, and NFS. All work well, within their respective
limitations - but of those, only ftp (and some NFS applications) support
resuming large files halfway.
You could use split, or something similar to
#!/bin/sh
SCRIPT="BLOCKNO=0;
do {
BYTES=\`dd bs=65536 count=1 of=$3.newpart |
sed -ne "s/\([0-9]*\) bytes transferred in.*/\1/p" ||
exit 1\`;
if [ \$BYTES -gt 0 ]; then
dd bs=65536 if=$3.newpart of=$3.part seek=\$BLOCKNO \
|| exit 1;
BLOCKNO=\$(( \$BLOCKNO + 1))
else
break;
fi
} while true;
rm $3.newpart;
mv $3.part $3"
dd if=$1 bs=65536 | ssh $2 sh -c "$SCRIPT"
but with a little protection against symlinks, and less bugs (the above
is untested!).
Joachim
