On Fri, 4 Jan 2002, Richard Atterer wrote:
> On Thu, Jan 03, 2002 at 04:34:52PM +0100, Josip Rodin wrote:
[..]
> > It seems the best thing would be to decide finally if we're going to
> > push jigdo or not. We can't have it recommended and mark it
> > <strong>beta</strong>, that's an oxymoron, at least in Debian. :)
Start pushing now. As far as I'm concerned, jigdo has just left beta stage.
(That's the jigdo scheme, not necessarily jigdo 0.6.1 ;-)
PROUDLY PRESENTING: jigdo-port - jigdo-lite "enhanced" - jigdo-lite2win
jigdo-port: highly portable subset of jigdo-file, compiles with any C compiler
jigdo-lite 2.0: enhanced & POSIX compatible - try it and you'll like it!
jigdo-lite2win (2 is for "to"; pun intended): jigdo-lite for M$Win
Get your CD images the smart way:
http://cdimage.debian.org/~costar/jigdo/
Available at this moment:
- the old 2.2 rev4 images
- the new 2.2 rev5 PRERELEASE-TESTING images -> please test & report!
- the revX-to-rev5 update CDs (under "Unofficial images")
[You can stop reading now if you aren't interested in some of my ideas &
technical details. Just go get those CDs! :-]
Short history of jigdo-port:
It all started when I had a little spare time and decided to take a look at
jigdo (0.6.1). It didn't compile on my potato box, 'cause it requires libdb3
and sstream.h. libdb3 was out-configurable, but replacing sstream by strstream
did compile but produced weird output in the list files. So compiled -static
on the sid chroot which did the trick.
Running it with jigdo-lite ("1.0" as --user-agent) worked nicely, and I took
a little closer look at the script. Now I always look at scripts with pico,
and the first tweak this time was the "clear" in hrule. And then the "page
headers" to be displayed under the "titlebar". That looked much better indeed.
But now the mirror selection (scrolling off-screen, cut&paste) didn't fit in
the new "display paradigm". A long time ago I'd already implemented a "list
selector" with numbers, so I took that idea and added a "scroll lock", which
needed fmt which I had just discovered. listinput() was born.
So, I had a nice mirror chooser. But I still needed to download the .jigdo
myself, while a nice listinput was available. Two hours later, I had designed
& implemented a quite simple yet powerful menu system in shell code on only
two A5 sheets (single-sided; I can write really small if needed ;-). This
worked beautifully and is pretty much unchanged since then.
I was quite happy with this, until I thought about another project that
involved portable/POSIX shell scripts and realised I had access to quite a few
"strange" machines I could test on at the Univ. So tested. Error. Fix. Test.
Another error. Fix. Test. Repeat. And of course every time I thought: "this
should be the last one," which only got true after a day or two.
I'd tested all these "ancient" platforms with a ": > jigdo-file; chmod a+x
jigdo-file". So at after a while I had a script that actually worked on most
systems but only to the point where jigdo-file was needed. In the process, I
had compiled wget on most systems, which always worked (except sometimes wget
couldn't resolve). But, frustratingly enough, I couldn't compile jigdo-file on
any system except my sid chroot. In other words, you first have to install
woody/sid, and only then you can download the potato images...
At that point I almost decided to feed my jigdo-lite changes back to Richard
and let him solve the jigdo-file issue. But after a little studying of the
jigdo docs (which are really excellent), I couldn't stop myself from thinking:
"well, it can't possibly be that hard to implement this in C." And I could use
a little zlib experience for yet another project, so I started coding.
And it really wasn't that hard. I estimate I only used some 20-25 hours, in
between the family events/visits from Dec 25 to Jan 1. Et voila jigdo-port,
which finally allowed me to download CD images "natively" on my potato boxes,
the 1992 SunOS/Sparc, the 1994 IRIX/mips and all other assorted machines.
But of course that wasn't yet enough. "If jigdo-lite is portable, it should
also run on Cygwin." And indeed, it does. Combined with the jigdo-lite-
compiled-for-Windows, this is an excellent combination, and it just works like
everywhere else. jigdo-lite2win was born, which is really nothing more than a
re-packaging of already-available parts.
End of "short" history. Read the various READMEs for a few more details.
Now for the future. I don't plan to do anything to jigdo-port/lite/lite2win
any more (unless I've made some stupid mistake somewhere) except fanatically
using it. It's my idea that -port and -lite get merged into the "official"
jigdo distribution. Please NOT hidden in some contrib/ dir, _many_ people need
it. Maybe also -lite could better be placed in it's own top-level dir, along
with its README and myprintf.c, and not in scripts/ along with all those
things that "end users" aren't interested in.
Mainly to Richard (already mentioned in the READMEs as maintainer -- unless
you don't want that ;-) : be _very_ careful with -port and especially -lite.
EVERYTHING has a good reason, even if that usually isn't documented.
-lite now works everywhere, and I'd very much like it staying that way.
So don't change it, or only in a way that's used (& tested) elsewhere in the
script.
I noted that you apparently like converting tabs to spaces, beware with -lite
since several tabs ARE significant there.
-port comes with the jull zlib-1.1.3 source, the good reason being that it was
only available "pre-installed" (by our excellent sysadmin) on one of the many
systems I tested on. Compiling a shared lib can be quite complex on many
ancient platforms; the corrent one `make' command links statically which works
great everywhere.
Short-term future: You can take over the online menu stuff; look in my mess
called public_html/jigdo on open how I did things. (sed comes in handy
sometimes ;-) I'll symlink my menu.jigdo (hardcoded in jigdo-lite) to your
version when you're ready. I'll handle the rev5 testing->official transition
so you can see how I'm thinking things should be done. My
public_html/jigdo/rev5gedoe has the details on how I generated the
rev5-testing stuff, including updated jigdo-cache.db.
On the future of the electronic Debian CD image distribution: I propose that
cdimage.debian.org will stop offering the CD images via rsync, as soon as
possible. Jigdo does a _much_ better job: the biggest template is only 8 MB
(alpha binary-1) and offering the templates via HTTP shouldn't produce any
significant load.
Mirrors can mirror the templates with wget. If a mirror admin has the
space/bandwidth, a simple script (print-missing>list, make-image
--files-from=list) can convert them quickly to full images (under 10 minutes
per image on my K6/450Mhz) when a local full Debian FTP mirror is available.
Handling mirrored templates with the current jigdo-lite is quite easy: a
cronjob on cdimage.debian.org that rewrites the .jigdos automatically every
minute pointing the Template= to another mirror. When the jigdo/template
become available on all Debian mirrors, jigdo-lite can be changed to have the
mirror selection before the downloading of the jigdo/template.
So for end users we'll have two download methods: jigdo and HTTP/FTP.
First-tier mirrors (mirroring from cdimage.d.o) can only use jigdo.
The Pseudo-Image Kit and rsync download/mirroring will be discontinued.
I've heard some concerns about the images being no longer available whenever a
new revision/release is released. Currently, this is true for the jigdo scheme
because the old/replaced files will vanish from the FTP mirrors after 3 or 4
days.
The real solution for that problem is of course convincing the mirror
maintainers that those files should _NOT_ vanish until the new CD images have
been released. (This doesn't require more disk space, since that space was in
use long before in proposed-updates, a few more weeks shouldn't be an issue.)
Another solution is regenerating the templates after the files have vanished.
Templates will probably get >20MB in that case. That's quite trivial, that's
why I didn't try that this time ;-) But also, it won't help people that
started downloading in the time between vanishing and regenerating, since they
will only need a few files. At least jigdo-port can't handle pasting the
contents of a new template into an existing image. So what I've done for the
currently-still-available-via-jigdo 2.2 rev4 images is extracting the needed
files directly from the .iso's and putting them in my public_html on
cdimage.d.o, and rewriting the .jigdo files. This works great.
If you're interested, look in
http://cdimage.debian.org/~costar/jigdo/22rev4-complete/
First run makemissing, then getfiles, finally newjigdo. Note that the scripts
need some customisation.
And of course the old images will still be available via HTTP/FTP, so keeping
the jigdo working might not even be strictly necessary.
You'll also see that I've radically changed the Filenames that are suggested
by the .jigdos. Since there's no need any longer to keep these names the same
for mirroring purposes, I figured it wouldn't hurt to make them more
descriptive. (It certainly helps me telling all these different images on my
systems apart ;-) Also for new users, anything is better than
'binary-i386-1.iso'. After a few weeks, one doesn't know any more what that
means.
Note that this suggested name is not obligatory; mirrors can still derive the
"old-style" name from the .jigdo and/or .template names.
For the people wanting to write their own menus: no use aligning anyting in
the Options= lines vertically. This won't look good on anything
non-Linux/Windows; emulate it by defining one of the "poor fmt substitutes" at
the beginning of jigdo-lite.
Finally one technical issue. Richard: I really don't know why you introduced
that quoting mess in the .jigdo format specs. The only reason I can think of
is that you want to support #comments after any option. The big problem is:
this is completely IMPOSSIBLE to parse in shell code (even if bashisms would
be allowed). If I were you, I'd drop the comments-after-options support right
now, and change the definition to
line := <startofline><label>=<value><newline>
<label> doesn't contain an =
label := everything (including space/tab) between startofline and first =
value := everything between first = and newline
This is easily parsable and can do everything you want. Comments can be on
lines of their own. These files are intended to be modified by experienced
users only, so no need to include space/tab-strippers, just be careful when
editting.
You can define special quoting only for selected fields, if you really need
that. For example in the mirrors list: "the entire <value> will be displayed
in the selection list, but the first space/tab marks the end of the URL."
(URLs can't have spaces.)
Oh, and jigdo-lite in jigdo-lite2win is called jigdo-lite.sh, because
otherwise when downloaded separately, Internet Explorer will call it
jigdo-lite.txt and that won't work.
Well, that's all I can think of at the moment. Quickly back to my thesis work
now, it's been far too long already ;-)
Regards,
Anne Bezemer
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
