On Mon, Mar 8, 2010 at 10:52 PM, Theo de Raadt <dera...@cvs.openbsd.org> wrote:
> No, that is not why.
>
> When snapshots go out *very quickly*, sometimes they overlap, and some
> ftp servers have a mix of old and new parts.
>
> There's no way to solve this without making the snapshot mirroring
> atomic. There are discussions about it..
I have written some simple shell scripts to track the status of the
ftp snapshot mirrors.
Basically they retrieve the SHA256 files, listings of the snapshot
installation file sets and listings of the packages. These listings
get tagged with a identifier to differentiate them from the
ftp.openbsd.org ones. The previously retrieved listings are archived,
so the new listings can be compared with the old ones.
For example
Current date : 2009-11-30_22.11_UTC
NOW date : 2009-11-30_22:10_UTC
PREV date : 2009-11-29_20:12_UTC
--------------- NOW ------------------------
2162 Nov 29 16:06:57 2009 SHA256-vienna
2162 Nov 29 16:06:57 2009 SHA256-stockholm
2162 Nov 29 16:06:57 2009 SHA256-nluug
2162 Nov 30 18:28:41 2009 SHA256
--------------- PREV -------------------
2162 Nov 28 06:43:08 2009 SHA256-vienna
2162 Nov 28 06:43:08 2009 SHA256-nluug
2162 Nov 29 16:06:57 2009 SHA256-stockholm
2162 Nov 29 16:06:57 2009 SHA256
--------------- X NOW ----------------------
15527631 Nov 29 08:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz
--------------- X PREV ----------------------
15527631 Nov 29 08:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz
--------------- PKG NOW --------------------
121808 Nov 29 12:12 9libs-1.0p5.tgz
9981 Nov 29 12:12 9menu-1.7p0.tgz
.... [snip] ...............
104955 Nov 29 12:22 zziplib-0.13.52.tgz
125760 Nov 29 12:22 zzuf-0.12.tgz
--------------- PKG PREV --------------------
121824 Nov 21 11:34 9libs-1.0p5.tgz
9981 Nov 21 11:34 9menu-1.7p0.tgz
.... [snip] ...............
104999 Nov 21 11:43 zziplib-0.13.52.tgz
125751 Nov 21 11:43 zzuf-0.12.tgz
This shows that ftp.openbsd.org's SHA256 is new, while the Stockholm, Vienna
and ftp.nluug.nl mirrors still have an older version.
The x* install sets on ftp.openbsd.org haven't changed, but a new set
of packages from 29th of Nov
have replaced the old ones from Nov 21st.
Another script compares the contents of the SHA* files and if they
match 'misuses' sdiff' to
show a side by side comparision:
1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org & stockholm ---
97834 Nov 29 08:06 INSTALL.i386 | 97834 Nov 29
15:06 INSTALL.i386
22356 Nov 29 08:06 INSTALL.linux | 22356 Nov 29
15:06 INSTALL.linux
2162 Nov 29 08:06 SHA256 | 2162 Nov 29 15:06 SHA256
49042398 Nov 29 08:06 base46.tgz | 49042398 Nov 29
15:06 base46.tgz
7490469 Nov 29 08:06 bsd | 7490469 Nov 29 15:06 bsd
7509916 Nov 29 08:06 bsd.mp | 7509916 Nov 29 15:06 bsd.mp
6234592 Nov 29 08:06 bsd.rd | 6234592 Nov 29 15:06 bsd.rd
6352896 Nov 29 08:06 cd46.iso | 6352896 Nov 29 15:06 cd46.iso
44980 Nov 29 08:06 cdboot | 44980 Nov 29 15:06 cdboot
2048 Nov 29 08:06 cdbr | 2048 Nov 29 15:06 cdbr
3012608 Nov 29 08:06 cdemu46.iso | 3012608 Nov 29
15:06 cdemu46.iso
91072515 Nov 29 08:06 comp46.tgz | 91072515 Nov 29
15:06 comp46.tgz
516738 Nov 29 08:06 etc46.tgz | 516738 Nov 29
15:06 etc46.tgz
1474560 Nov 29 08:06 floppy46.fs | 1474560 Nov 29
15:06 floppy46.fs
1474560 Nov 29 08:06 floppyB46.fs | 1474560 Nov 29
15:06 floppyB46.fs
1474560 Nov 29 08:06 floppyC46.fs | 1474560 Nov 29
15:06 floppyC46.fs
2618027 Nov 29 08:06 game46.tgz | 2618027 Nov 29
15:06 game46.tgz
269 Nov 29 08:06 index.txt | 269 Nov 29
15:06 index.txt
249757696 Nov 29 08:06 install46.iso | 249757696 Nov 29
15:06 install46.iso
8469749 Nov 29 08:06 man46.tgz | 8469749 Nov 29
15:06 man46.tgz
2936230 Nov 29 08:06 misc46.tgz | 2936230 Nov 29
15:06 misc46.tgz
53532 Nov 29 08:06 pxeboot | 53532 Nov 29 15:06 pxeboot
15527631 Nov 29 08:02 xbase46.tgz | 15527631 Nov 29
15:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz | 68103 Nov 29
15:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz | 35215028 Nov 29
15:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz | 19874775 Nov 29
15:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz | 2950471 Nov 29
15:02 xshare46.tgz
3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
For the over 6000 snapshot packages, only the unique dates are compared:
Unique dates from Unique dates from
./NOW/latest_pkg: | ./NOW/latest_pkg-nluug:
=============================== ===============================
1 11-21 11:34 | 1 11-21 18:34
2 11-21 11:35 | 2 11-21 18:35
3 11-21 11:36 | 3 11-21 18:36
4 11-21 11:37 | 4 11-21 18:37
5 11-21 11:38 | 5 11-21 18:38
6 11-21 11:39 | 6 11-21 18:39
7 11-21 11:40 | 7 11-21 18:40
8 11-21 11:41 | 8 11-21 18:41
9 11-21 11:42 | 9 11-21 18:42
10 11-21 11:43 | 10 11-21 18:43
11 11-28 04:05 | 11 11-27 11:05
=============================== ===============================
Last entry: Last entry:
11-28 04:05 index.txt 456441 | 11-27 11:05 index.txt 456441
All files with last date: All files with last date:
"11-28 04:05" | "11-27 11:05"
=============================== ===============================
11-28 04:05 index.txt 456441 | 11-27 11:05 index.txt 456441
=============================== ===============================
The documentation of these scripts are at
http://siralas.nl/OBSDsnapshot-tracking.html
The scripts and the installation Makefile at
http://siralas.nl/OBSDsnapshot_tracking-1.05.tgz
Have fun ;)
Adriaan
