Hello,
I am questioning whether it is possible to make a local mirror that is
consistent, i.e. that doesn't contain a mixup of versions due to
non-atomic rsync (mirrors trying to sync from what I imagine is a
master, or between each other, and my server trying to sync from one
of the mirrors). At the moment, my local copy seems corrupted
(setup-x86.exe croaks on signature error and refuses to go ahead), and
I haven't found another mirror to rsync from that would repair the
situation. Several rsyncs from different mirrors lead to the same
situation: an apparently corrupted local repo.
Details
-------
Being behind a (slowish) firewall, I set up a local cygwin mirror on a
linux server.
I let an rsync-based script on crontab eat the slow connection, once
daily, and most of the time rsync does its thing with minimal impact
and all is well.
However, I've bumped today into what seems to be a mismatch of
signatures and versions. I've tried several mirrors as the source for
my local copy, but I've been unable to find one that let's the client
cygwin installer work.
Instead, when running setup-x86.exe (the latest one, always downloaded
by the same script as the one running rsync), I get a "Mirror error:
Setup.ini signature (...) setup.bz2.sig from (...) failed to verify".
Thinking a bit more about it, I wonder what mechanisms, if any, are in
place to provide consistency and atomicity of the process? Is there a
special time of the day when the "master copy" is frozen, the
signatures and md5sum are calculated, and a new version is released
for the mirrors to pick up?
Also, if the mirrors are simply updated via rsync, how do I make sure
that I am not myself rsyncing when they are in the midst of it?
I was curious about timestamps and md5sums on various mirrors (I
looked only in the US ones --I put the one-liner to do that as
post-scriptum). And indeed, I have found that different mirrors have
different signature files, and with various timestamps.
For example, as of 20140910-001532UTC, I find the following timestamps
for various setup.bz2.sig:
- 20140908-231037UTC mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.bz2.sig
- 20140909-191037UTC mirrors.syringanetworks.net_cygwin/x86/setup.bz2.sig
- 20140909-191037UTC mirrors.xmission.com_cygwin/x86/setup.bz2.sig
- 20140909-201044UTC mirror.team-cymru.org_cygwin/x86/setup.bz2.sig
- 20140909-231038UTC rsync.osuosl.org_cygwin/x86/setup.bz2.sig
- 20140910-001035UTC mirrors.kernel.org_sourceware_cygwin/x86/setup.bz2.sig
Worse, their md5sum don't match:
# grep -n -E 'bz2.sig$; */x86/md5.sum | perl -ne 'chomp;s/:/
/g;@a=split;print join("\t",@a[2,0])."\n"'
f88c73ee59cf59f7f32ca4bad26376c1
mirrors.kernel.org_sourceware_cygwin/x86/md5.sum
15a0cca932d75390e01b54a46b2558e2
mirrors.syringanetworks.net_cygwin/x86/md5.sum
f449fdb08944b8563c0ad273069669f1
mirrors-usa.go-parts.com_mirrors_cygwin/x86/md5.sum
15a0cca932d75390e01b54a46b2558e2 mirrors.xmission.com_cygwin/x86/md5.sum
413c8532019a41a1b310482cad0d25bb mirror.team-cymru.org_cygwin/x86/md5.sum
6616150da358594eaa613032b66d2681 rsync.osuosl.org_cygwin/x86/md5.sum
Any suggestion or explanation?
Thanks!
~Pierre
PS: test all the signatures of the US-based rsync:// servers:
# mkcd ~/tmp/cygwin_signatures
# for i ($(perl -ne 's,/",",g;print "$1\n" while
(s,"rsync://(.*?)",,)' us_mirror_list.html)); do j=$i:gs,/,_; echo $i;
rsync -vaz --exclude 'release/' --progress --timeout 3600 rsync://$i/
$j/; done
# head -20 */x86/md5.sum
==> mirrors.kernel.org_sourceware_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb setup.bz2
f88c73ee59cf59f7f32ca4bad26376c1 setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 setup.ini
4b0a55de2687c537124139bd569328ae setup.ini.sig
==> mirrors.syringanetworks.net_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb setup.bz2
15a0cca932d75390e01b54a46b2558e2 setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 setup.ini
d2d94f4e1734e896af4efa44e4c6654f setup.ini.sig
==> mirrors-usa.go-parts.com_mirrors_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb setup.bz2
f449fdb08944b8563c0ad273069669f1 setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 setup.ini
9ff8b9cfb0e29b9960ae918bb68a4dd6 setup.ini.sig
==> mirrors.xmission.com_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb setup.bz2
15a0cca932d75390e01b54a46b2558e2 setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 setup.ini
d2d94f4e1734e896af4efa44e4c6654f setup.ini.sig
==> mirror.team-cymru.org_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb setup.bz2
413c8532019a41a1b310482cad0d25bb setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 setup.ini
212b298441fdf11ee45ef3841dd38b62 setup.ini.sig
==> rsync.osuosl.org_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb setup.bz2
6616150da358594eaa613032b66d2681 setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 setup.ini
75dd5f39d40c255951daf0ab5b865333 setup.ini.sig
# md5sum */x86/setup*
a7f25f8d488a30b68cf1c08f66ca35cb
mirrors.kernel.org_sourceware_cygwin/x86/setup.bz2
f88c73ee59cf59f7f32ca4bad26376c1
mirrors.kernel.org_sourceware_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936
mirrors.kernel.org_sourceware_cygwin/x86/setup.ini
4b0a55de2687c537124139bd569328ae
mirrors.kernel.org_sourceware_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb
mirrors.syringanetworks.net_cygwin/x86/setup.bz2
15a0cca932d75390e01b54a46b2558e2
mirrors.syringanetworks.net_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936
mirrors.syringanetworks.net_cygwin/x86/setup.ini
d2d94f4e1734e896af4efa44e4c6654f
mirrors.syringanetworks.net_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.bz2
f449fdb08944b8563c0ad273069669f1
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.ini
9ff8b9cfb0e29b9960ae918bb68a4dd6
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb mirrors.xmission.com_cygwin/x86/setup.bz2
15a0cca932d75390e01b54a46b2558e2 mirrors.xmission.com_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 mirrors.xmission.com_cygwin/x86/setup.ini
d2d94f4e1734e896af4efa44e4c6654f mirrors.xmission.com_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb mirror.team-cymru.org_cygwin/x86/setup.bz2
413c8532019a41a1b310482cad0d25bb mirror.team-cymru.org_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 mirror.team-cymru.org_cygwin/x86/setup.ini
212b298441fdf11ee45ef3841dd38b62 mirror.team-cymru.org_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb rsync.osuosl.org_cygwin/x86/setup.bz2
6616150da358594eaa613032b66d2681 rsync.osuosl.org_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936 rsync.osuosl.org_cygwin/x86/setup.ini
75dd5f39d40c255951daf0ab5b865333 rsync.osuosl.org_cygwin/x86/setup.ini.sig
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
