Hi, I have been having a very annoying problem with fetch and the ports distfiles fetching. Living in South Africa internet access is not what it could be, as such I have to use a proxy to access the internet. On to the problem:
When fetch tries to resume from an ftp site that does not support resuming (or
fetch lacks the ability in these specific cases, or the proxy + fetch
faulted) it appends to the file but restarts the transfer from 0. This
results in a file that is too big but does contain a complete copy of the
file. A solution is simple:
es = Expected size of the file
as = Actual size of the file
f = File name
# dd if=f of=f~ bs=$((as - es)) skip=1
# mv f~ f
This is very inconvenient and a problem when trying to download files in
batch. One gets interrupted and stops everything else. As such I looked
into fetch.c and tried to fix the problem. See below for my patch. A
problem is that how to handle the -R flag (no overwrite file flag). I have
three solutions:
1) Ignore flag and overwrite the file anyway
2) Do not overwrite and abort
3) Specify above behaviour with an additional flag
Given that the ports and having many mirrors it is possible to find a site
that does support resuming, and thus will save in bandwidth (important for
me). However it is possible that there are no alternatives or all sites do
not support resume.
In the patch the commented out section is solution 2, the implemented version
is solution 1.
Any advice will be welcome, once a satisfactory solution has been achieved I
will submit a PR with the patch?
Thank you
David
P.S. I've been using FreeBSD for a year or two however I am very new at
writing patches for it, this will be my first so all help and mentoring will
be very welcome.
Patch file:
--- usr.bin/fetch/fetch.c.orig 2007-06-07 09:17:18.000000000 +0200
+++ usr.bin/fetch/fetch.c 2007-06-07 09:18:39.000000000 +0200
@@ -510,8 +510,36 @@
(intmax_t)sb.st_size, (intmax_t)us.size);
goto failure;
}
+// /* check fetched offset is as expected */
+// if (url->offset != sb.st_size) {
+// if (R_flag) {
+// /* unable to resume */
+// warnx("%s: unexpected offset, "
+// "expected %jd bytes, got %jd "
+// "bytes", (intmax_t)sb.st_size,
+// (intmax_t)url->offset);
+// goto failure_keep;
+// } else if (url->offset != 0)
+// /* if not we need to restart from scratch */
+// goto restart;
+// }
+// if ((of = fopen(path, "a")) == NULL) {
+// warn("%s: fopen()", path);
+// goto failure;
+// }
+// /* check feteched offset is either as expected or 0 */
+ if (url->offset != sb.st_size || url->offset != 0)
+ /* if not we need to restart from scratch */
+ goto restart;
+ /*
+ * NOTE: R_flag is ignored and local file is over
+ * written *not* deleted as it is the only way
+ * to resume transfer of this file from this
+ * URL.
+ */
/* we got it, open local file */
- if ((of = fopen(path, "a")) == NULL) {
+ if ((of = url->offset == 0 ? fopen(path, "w") :
+ fopen(path, "a")) == NULL) {
warn("%s: fopen()", path);
goto failure;
}
@@ -536,6 +564,7 @@
goto success;
}
+restart:
if (of == NULL) {
/*
* We don't yet have an output file; either this is a
pgp1L79Bm48wE.pgp
Description: PGP signature
