Wido - in that case should we disable metadata re-generating cronjobs for both rpm/deb repos? We only need to regenerate repo metadata on a new release. There's no need to do this via cron for official releases.
Regards. ________________________________ From: Wido den Hollander <w...@widodh.nl> Sent: Monday, February 15, 2021 21:30 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Rohit Yadav <rohit.ya...@shapeblue.com> Subject: Re: RPM Repository Metadata On 15/02/2021 09:51, Rohit Yadav wrote: > Hi Nathan, > > Thanks for reporting, I've been managing the rpm builds/repos on the server > and I wasn't aware of this issue. > I checked and found there's a hourly cron job that updates rpm repo metadata > using: > > createrepo --update --workers 1 --baseurl <other options/paths> > > > Based on your suggestion, I've changed the script to include: "createrepo > --update --retain-old-md <rest of the args...>". > > @Wido Hollander<mailto:w...@pcextreme.nl> @Gabriel Beims > Bräscher<mailto:gabr...@pcextreme.nl> - any reason why we have the cron job > to update repo metadata? No, I think it's just an oversight. It was setup and I don't think it was very well thought of. The CRON is a very simple Shell script which probably can use some attention. Wido > > Regards. > > ________________________________ > From: Nathan McGarvey <nathanmcgar...@gmail.com> > Sent: Sunday, February 14, 2021 08:32 > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > Subject: RPM Repository Metadata > > To whom this reaches (@widodh, perhaps?): > > First of all, thank you for building binary distributions (rpm, deb) > of CloudStack. > > I am attempting to create a downstream rsync mirror of the > http://download.cloudstack.org/ centos/rhel repos. (Namely, centos and > systemvm) and noticed two oddities: > > 1. The frequency with which the metadata is being rebuilt is > astronomical. E.g. > http://download.cloudstack.org/centos/8/4.15/repodata/ looks like it is > being fully rebuilt every hour, though the RPMs contained within haven't > been updated in over a month. Was this supposed to have a > --retain-old-md or --retain-old-md-by-age flag? The default for things > like RHEL 8 is 48 hours for metadata expiry, so re-generating the entire > repo every hour can (and does) cause caching issues. > > 2. The metadata contained in > http://download.cloudstack.org/centos/8/4.15/repodata/repomd.xml makes > it virtually impossible to mirror since it points the <location> tag at > http://cloudstack.apt-get.eu/ > a. Most RPM repos (E.g. The CentOS official ones, just point the > <location> tag to the repodata/<hash>> of the data type without external > links via relative URI. (See > http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/repodata/repomd.xml > , for example.) > b. Putting the xml:base in there effectively makes it not a > mirror since anyone pointed at their local mirror will actually redirect > to whatever xml:base is set to. > > FWIW: It looks like cloudstack.apt-get.eu and > download.cloudstack.org are *probably* the same host, which likely means > the xml:base being set at all may not actually be doing anything useful. > > > Please pardon my ignorance if these technical configurations are > intentional and have already been discussed as I am new poster to this > thread. > > > Thanks, > -Nathan McGarvey > > rohit.ya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > rohit.ya...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
