On Fri, Jan 18, 2002 at 01:59:11PM +0100, J.A. Bezemer wrote:
> As far as I'm concerned, jigdo has just left beta stage. (That's the
> jigdo scheme, not necessarily jigdo 0.6.1 ;-)
Hm, for *me* jigdo will be finished the moment my mother can download
a CD image without me giving her any instructions! ;-]
BTW, Anne (or anybody else who is interested), should you have more
spare time in the future, here are some fun projects:
- Hack debian-cd to allow output of .jigdo/.template files without
saving a temporary image to disc. A must for DVD jigdo generation.
- Add DVD support to debian-cd (trivial AFAIK?)
- Hack mkhybrid to output .jigdo/.template instead of raw ISO9660
data. This would make the template generation a lot faster -
imagine daily "testing" DVD images!
- A CGI script (or mini web server, or Apache module...:) which
assembles the CD image on the fly from .jigdo, .template and a
local mirror. Essentially, this allows mirror admins to offer
direct HTTP downloads without the need to store the complete images
locally. (HTTP/1.1 ranges support would be cool.)
> PROUDLY PRESENTING: jigdo-port - jigdo-lite "enhanced" -
> jigdo-lite2win
Great work, Anne! After a look through all this, I can attest that
it's a lot nicer than my quick "just knock up something that works"
jigdo-lite.
[snip]
> It didn't compile on my potato box, 'cause it requires libdb3 and
> sstream.h. libdb3 was out-configurable, but replacing sstream by
> strstream did compile but produced weird output in the list files.
/me awards himself the Portable Programming of the Month award. :-/
OH NO! It is definitely my intention to make jigdo-file compilable on
potato! After I noticed that sstream worked with GCC 2.95 under
testing, I started to use it. I didn't know that the GCC 2.95 in
potato does not cope with it.
I'll try to remove the dependency on sstream soon. As for the other
C++ features that many C++ compilers out there probably don't support
yet: The C++ ISO standard has been out for three years, I can't be
bothered to enter "write once, test everywhere" mode. GCC 2.95 or 3 is
available for all the important arches...
[snip]
> Now for the future. I don't plan to do anything to
> jigdo-port/lite/lite2win any more (unless I've made some stupid
> mistake somewhere) except fanatically using it. It's my idea that
> -port and -lite get merged into the "official" jigdo distribution.
> Please NOT hidden in some contrib/ dir, _many_ people need it. Maybe
> also -lite could better be placed in it's own top-level dir, along
> with its README and myprintf.c, and not in scripts/ along with all
> those things that "end users" aren't interested in.
That's fine with me, as soon as the remaining issues (see below) are
resolved! But hopefully you won't just "dump" this on me and then
disappear? :) Your shell scripting expertise and test machines will be
needed for future modifications to the script!
But I'd rather not include the zlib source in the jigdo source
distribution - doing that increases its size quite a bit, also it's
easy for people to download it...
Should jigdo-port.c use the configure mechanism?
> Mainly to Richard (already mentioned in the READMEs as maintainer --
> unless you don't want that ;-) : be _very_ careful with -port and
> especially -lite. EVERYTHING has a good reason, even if that usually
> isn't documented.
That's what I really hate about shell scripting (and why I'd prefer to
leave future changes to the script to you) - making any non-trivial
shell script portable is a nightmare!
> Short-term future: You can take over the online menu stuff; look in
> my mess called public_html/jigdo on open how I did things.
The menu stuff is the one big thing where I don't like what you've
done. The idea I had was eventually to have just *one* jigdo file per
Debian CD release. That file would contain info on all the CDs plus
the mirror list. IMHO this is plain and simple, and easy to understand
for users. There's no reason to spread the info across a menu.jigdo,
mirrors.jigdo, binary*.jigdo; that only increases the complexity.
(So why don't I offer just /one/ .jigdo? Laziness... ;-)
The "Info" label in the [Jigdo] section would contain an introductory
text along the lines of "choose the correct arch, you only need one
out of binary-1 and binary-1-NONUS", etc. When the user selects one
image, that image's "Info" label would be shown.
Your menu structure also has the problem that it's a bit too flexible
- making a GUI app parse and display it wouldn't be impossible, but
I'd rather keep things simple and have just one list (generated from
the "ShortInfo" labels of all the "[Image]" sections) showing all the
available images.
Another, minor point: Hard-coding the initial .jigdo URL into the
script doesn't seem right to me. Of course you can always download to
a file and supply that, but that will not be obvious to new users who
want to download non-Debian releated stuff.
> On the future of the electronic Debian CD image distribution: I
> propose that cdimage.debian.org will stop offering the CD images via
> rsync, as soon as possible.
We discussed this before - essentially, despite being flattered, I
still disagree. ;) Let rsync and HTTP/FTP mirrors co-exist with jigdo
throughout "woody==stable" - that way, there's enough time to improve
jigdo and fix bugs without being flooded by angry users.
> Jigdo does a _much_ better job: the biggest template is only 8 MB
> (alpha binary-1) and offering the templates via HTTP shouldn't
> produce any significant load.
Beware - you need to include debian-keyring.tar.gz (~4MB) in the
.template, because it changes quite often on the FTP server! See the
regexp in make-templates for other stuff I've chosen to ignore during
scanning. (Or alternatively, you need to run your check for missing
files daily or so from a cron script.)
> Handling mirrored templates with the current jigdo-lite is quite
> easy: a cronjob on cdimage.debian.org that rewrites the .jigdos
> automatically every minute pointing the Template= to another mirror.
Yuk! :-)
No, what I *intend* to do eventually is:
Template=Debian:cd-images/templates/foobar/worble.template
...just like for all the other URIs in the .jigdo file!
> When the jigdo/template become available on all Debian mirrors,
> jigdo-lite can be changed to have the mirror selection before the
> downloading of the jigdo/template.
Yes.
> So for end users we'll have two download methods: jigdo and
> HTTP/FTP. First-tier mirrors (mirroring from cdimage.d.o) can only
> use jigdo. The Pseudo-Image Kit and rsync download/mirroring will be
> discontinued.
There is no compelling reason ATM to discontinue rsync, so IMHO we
should not do it. Of course, when DVD images become available, there
won't be any way of providing them as raw images.
> I've heard some concerns about the images being no longer available
> whenever a new revision/release is released. Currently, this is true
> for the jigdo scheme because the old/replaced files will vanish from
> the FTP mirrors after 3 or 4 days.
>
> The real solution for that problem is of course convincing the
> mirror maintainers that those files should _NOT_ vanish until the
> new CD images have been released. (This doesn't require more disk
> space, since that space was in use long before in proposed-updates,
> a few more weeks shouldn't be an issue.)
Right. (It does need a bit more disk space if you want to offer
"testing" images via jigdo.) For the moment though, I plan to add
support for a fallback server to download from if the selected mirror
doesn't offer a file anymore. That fallback download uses the required
file's md5sum as the leafname (cf "MD5Sum:<checksum>" in
jigdo-file(1)).
Using the md5sum is preferable to just keeping the old files, because
it also deals with the condition that a file can be *updated* instead
of deleted during the lifetime of a jigdo image.
Looks like Attila Nagy will be offering something like that soon.
Maybe cdimage should, too.
> So what I've done for the currently-still-available-via-jigdo 2.2
> rev4 images is extracting the needed files directly from the .iso's
> and putting them in my public_html on cdimage.d.o, and rewriting the
> .jigdo files. This works great.
...as long as you still *have* the original images!
> You'll also see that I've radically changed the Filenames that are
> suggested by the .jigdos. Since there's no need any longer to keep
> these names the same for mirroring purposes, I figured it wouldn't
> hurt to make them more descriptive.
Agreed in principle, but
- The names should be identical to those of the .iso files, else it's
a bit confusing. So once we change the jigdo names, the .iso names
should change as well.
- Let's not do this change now, in the middle of 2.2, let's switch
over to a new naming scheme for 3.0
> Finally one technical issue. Richard: I really don't know why you
> introduced that quoting mess in the .jigdo format specs.
:-)
It's because sooner or later there will be support for switches after
the label values. Things like different --priority values for
different mirrors, --referer to allow you to upload jigdo stuff to
geocities.com ;), --jpeg-steg to extract data out of a JPEG, --decrypt
with a passphrase, --make-coffee, --dominate-world, ... endless
possiblities!
But yes, I had the same problems with this as you when I wrote
jigdo-lite. The parsing functionality is already present in
jigdo-file, so maybe it should just be made accessible for shell
scripts with "jigdo-file print-label MySection LabelName -j foo.jigdo".
Oh, I need to pick out a few things from your jigdo docs:
In jigdo-port.c:
| Copyright (C) 2002 J.A. Bezemer <[EMAIL PROTECTED]>
| Licensed under the GNU General Public License, version 2 exactly.
| I didn't take a single look at the official jigdo-file source except
| for the base64-but-why-`-'-and-`_' details.
| If you want an example of beautiful object oriented coding, see jigdo-file.
| If you want the harsh reality of fully-portable plain-C coding, look here.
LOL! Yes, yes, I'll at least make it compile on potato! :-)
In jigdo-lite.README:
| 14. After downloading a lot of files it says it still needs more.
|
| As it also says, try again with another Debian FTP mirror. Not all
| mirrors are equally up to date.
|
| If that doesn't work, chances are that the required files are not
| available at this time. Try again in a few days and you'll either see
| that the required files are available again, or that a new image has
| been created.
|
| In the latter case, you can probably still re-use the files that were
| downloaded in the incomplete image, simply because that image is
| logically correct. The missing files will show up just like they
| should, with the correct length, but will contain all zeros.
| After renaming the incomplete .tmp file (because that same name will
| be used for the new image), you can loop-mount it and point
| jigdo-lite's `Local files' question to it.
That's of course correct, but I'd consider it something for advanced
users only. <sigh> A better fix would be for "jigdo-file scan" to
recognize "x.template" and "x.iso.tmp" files when it sees them -
difficult...
Last words: I'm graduating from uni in two months. I'll be very busy
indeed, don't expect any major updates anytime soon. :-/
Cheers,
Richard
--
__ _
|_) /| Richard Atterer | CS student at the Technische | GnuPG key:
| \/¯| http://atterer.net | Universität München, Germany | 0x888354F7
¯ '` ¯
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
