Re: RPM Repository Metadata

[Rohit Yadav]

Hi Nathan,

Thanks for reporting, I've been managing the rpm builds/repos on the server and 
I wasn't aware of this issue.
I checked and found there's a hourly cron job that updates rpm repo metadata 
using:

createrepo --update --workers 1 --baseurl 


Based on your suggestion, I've changed the script to include: "createrepo 
--update --retain-old-md ".

@Wido Hollander @Gabriel Beims 
Bräscher - any reason why we have the cron job to 
update repo metadata?

Regards.


From: Nathan McGarvey 
Sent: Sunday, February 14, 2021 08:32
To: dev@cloudstack.apache.org 
Subject: RPM Repository Metadata

To whom this reaches (@widodh, perhaps?):

First of all, thank you for building binary distributions (rpm, deb)
of CloudStack.

I am attempting to create a downstream rsync mirror of the
http://download.cloudstack.org/ centos/rhel repos. (Namely, centos and
systemvm) and noticed two oddities:

1. The frequency with which the metadata is being rebuilt is
astronomical. E.g.
http://download.cloudstack.org/centos/8/4.15/repodata/ looks like it is
being fully rebuilt every hour, though the RPMs contained within haven't
been updated in over a month. Was this supposed to have a
--retain-old-md or --retain-old-md-by-age flag? The default for things
like RHEL 8 is 48 hours for metadata expiry, so re-generating the entire
repo every hour can (and does) cause caching issues.

2. The metadata contained in
http://download.cloudstack.org/centos/8/4.15/repodata/repomd.xml makes
it virtually impossible to mirror since it points the  tag at
http://cloudstack.apt-get.eu/
a. Most RPM repos (E.g. The CentOS official ones, just point the
 tag to the repodata/> of the data type without external
links via relative URI. (See
http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/repodata/repomd.xml
, for example.)
b. Putting the xml:base in there effectively makes it not a
mirror since anyone pointed at their local mirror will actually redirect
to whatever xml:base is set to.

FWIW: It looks like cloudstack.apt-get.eu and
download.cloudstack.org are *probably* the same host, which likely means
the xml:base being set at all may not actually be doing anything useful.


Please pardon my ignorance if these technical configurations are
intentional and have already been discussed as I am new poster to this
thread.


Thanks,
-Nathan McGarvey

rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

[GitHub] [cloudstack-documentation] Doni7722 commented on pull request #172: Update user-data.rst

[GitBox]

Doni7722 commented on pull request #172:
URL: 
https://github.com/apache/cloudstack-documentation/pull/172#issuecomment-779105074


   @onitake I did verify and can confirm that it works fine with 
`datasource_list: [ CloudStack, None ]` without `datasource`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-documentation] blueorangutan commented on pull request #189: Update kvm.rst

[GitBox]

blueorangutan commented on pull request #189:
URL: 
https://github.com/apache/cloudstack-documentation/pull/189#issuecomment-779137219


   Doc build preview: http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/189. 
(SL-JID-[S]17)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: RPM Repository Metadata

[Wido den Hollander]

On 15/02/2021 09:51, Rohit Yadav wrote:

Hi Nathan,

Thanks for reporting, I've been managing the rpm builds/repos on the server and 
I wasn't aware of this issue.
I checked and found there's a hourly cron job that updates rpm repo metadata 
using:

 createrepo --update --workers 1 --baseurl 


Based on your suggestion, I've changed the script to include: "createrepo --update 
--retain-old-md ".

@Wido Hollander @Gabriel Beims 
Bräscher - any reason why we have the cron job to update 
repo metadata?


No, I think it's just an oversight. It was setup and I don't think it 
was very well thought of.


The CRON is a very simple Shell script which probably can use some 
attention.


Wido



Regards.


From: Nathan McGarvey 
Sent: Sunday, February 14, 2021 08:32
To: dev@cloudstack.apache.org 
Subject: RPM Repository Metadata

To whom this reaches (@widodh, perhaps?):

 First of all, thank you for building binary distributions (rpm, deb)
of CloudStack.

 I am attempting to create a downstream rsync mirror of the
http://download.cloudstack.org/ centos/rhel repos. (Namely, centos and
systemvm) and noticed two oddities:

 1. The frequency with which the metadata is being rebuilt is
astronomical. E.g.
http://download.cloudstack.org/centos/8/4.15/repodata/ looks like it is
being fully rebuilt every hour, though the RPMs contained within haven't
been updated in over a month. Was this supposed to have a
--retain-old-md or --retain-old-md-by-age flag? The default for things
like RHEL 8 is 48 hours for metadata expiry, so re-generating the entire
repo every hour can (and does) cause caching issues.

 2. The metadata contained in
http://download.cloudstack.org/centos/8/4.15/repodata/repomd.xml makes
it virtually impossible to mirror since it points the  tag at
http://cloudstack.apt-get.eu/
 a. Most RPM repos (E.g. The CentOS official ones, just point the
 tag to the repodata/> of the data type without external
links via relative URI. (See
http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/repodata/repomd.xml
, for example.)
 b. Putting the xml:base in there effectively makes it not a
mirror since anyone pointed at their local mirror will actually redirect
to whatever xml:base is set to.

 FWIW: It looks like cloudstack.apt-get.eu and
download.cloudstack.org are *probably* the same host, which likely means
the xml:base being set at all may not actually be doing anything useful.


 Please pardon my ignorance if these technical configurations are
intentional and have already been discussed as I am new poster to this
thread.


Thanks,
-Nathan McGarvey

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
   
  

Can I setup AD zone

[Support Admin]

*Hi,*

Can I setup Advanced zone using private ip as public ip with tag vLAN? 
This is my testing environment. Please give me any network diagram for this.


--

*Thanks & Regards.*

Support Admin

Facebook  | Twitter 
 | Website 


116/1 West Malibagh, D. I. T Road

Dhaka-1217, Bangladesh

*Mob :* +088 01716915504

*Email :* support.ad...@technologyrss.com

*Web :* www.technologyrss.com 

Re: RPM Repository Metadata

[Rohit Yadav]

Wido - in that case should we disable metadata re-generating cronjobs for both 
rpm/deb repos? We only need to regenerate repo metadata on a new release. 
There's no need to do this via cron for official releases.


Regards.


From: Wido den Hollander 
Sent: Monday, February 15, 2021 21:30
To: dev@cloudstack.apache.org ; Rohit Yadav 

Subject: Re: RPM Repository Metadata



On 15/02/2021 09:51, Rohit Yadav wrote:
> Hi Nathan,
>
> Thanks for reporting, I've been managing the rpm builds/repos on the server 
> and I wasn't aware of this issue.
> I checked and found there's a hourly cron job that updates rpm repo metadata 
> using:
>
>  createrepo --update --workers 1 --baseurl 
>
>
> Based on your suggestion, I've changed the script to include: "createrepo 
> --update --retain-old-md ".
>
> @Wido Hollander @Gabriel Beims 
> Bräscher - any reason why we have the cron job 
> to update repo metadata?

No, I think it's just an oversight. It was setup and I don't think it
was very well thought of.

The CRON is a very simple Shell script which probably can use some
attention.

Wido

>
> Regards.
>
> 
> From: Nathan McGarvey 
> Sent: Sunday, February 14, 2021 08:32
> To: dev@cloudstack.apache.org 
> Subject: RPM Repository Metadata
>
> To whom this reaches (@widodh, perhaps?):
>
>  First of all, thank you for building binary distributions (rpm, deb)
> of CloudStack.
>
>  I am attempting to create a downstream rsync mirror of the
> http://download.cloudstack.org/ centos/rhel repos. (Namely, centos and
> systemvm) and noticed two oddities:
>
>  1. The frequency with which the metadata is being rebuilt is
> astronomical. E.g.
> http://download.cloudstack.org/centos/8/4.15/repodata/ looks like it is
> being fully rebuilt every hour, though the RPMs contained within haven't
> been updated in over a month. Was this supposed to have a
> --retain-old-md or --retain-old-md-by-age flag? The default for things
> like RHEL 8 is 48 hours for metadata expiry, so re-generating the entire
> repo every hour can (and does) cause caching issues.
>
>  2. The metadata contained in
> http://download.cloudstack.org/centos/8/4.15/repodata/repomd.xml makes
> it virtually impossible to mirror since it points the  tag at
> http://cloudstack.apt-get.eu/
>  a. Most RPM repos (E.g. The CentOS official ones, just point the
>  tag to the repodata/> of the data type without external
> links via relative URI. (See
> http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/repodata/repomd.xml
> , for example.)
>  b. Putting the xml:base in there effectively makes it not a
> mirror since anyone pointed at their local mirror will actually redirect
> to whatever xml:base is set to.
>
>  FWIW: It looks like cloudstack.apt-get.eu and
> download.cloudstack.org are *probably* the same host, which likely means
> the xml:base being set at all may not actually be doing anything useful.
>
>
>  Please pardon my ignorance if these technical configurations are
> intentional and have already been discussed as I am new poster to this
> thread.
>
>
> Thanks,
> -Nathan McGarvey
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>

rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

Re: RPM Repository Metadata

[Wido den Hollander]

On 16/02/2021 07:00, Rohit Yadav wrote:

Wido - in that case should we disable metadata re-generating cronjobs for both 
rpm/deb repos? We only need to regenerate repo metadata on a new release. 
There's no need to do this via cron for official releases.



I have disabled the CRON jobs for now for both RPM and DEB. Need to run 
them manually if we upload new packages.


Wido



Regards.


From: Wido den Hollander 
Sent: Monday, February 15, 2021 21:30
To: dev@cloudstack.apache.org ; Rohit Yadav 

Subject: Re: RPM Repository Metadata



On 15/02/2021 09:51, Rohit Yadav wrote:

Hi Nathan,

Thanks for reporting, I've been managing the rpm builds/repos on the server and 
I wasn't aware of this issue.
I checked and found there's a hourly cron job that updates rpm repo metadata 
using:

  createrepo --update --workers 1 --baseurl 


Based on your suggestion, I've changed the script to include: "createrepo --update 
--retain-old-md ".

@Wido Hollander @Gabriel Beims 
Bräscher - any reason why we have the cron job to update 
repo metadata?


No, I think it's just an oversight. It was setup and I don't think it
was very well thought of.

The CRON is a very simple Shell script which probably can use some
attention.

Wido



Regards.


From: Nathan McGarvey 
Sent: Sunday, February 14, 2021 08:32
To: dev@cloudstack.apache.org 
Subject: RPM Repository Metadata

To whom this reaches (@widodh, perhaps?):

  First of all, thank you for building binary distributions (rpm, deb)
of CloudStack.

  I am attempting to create a downstream rsync mirror of the
http://download.cloudstack.org/ centos/rhel repos. (Namely, centos and
systemvm) and noticed two oddities:

  1. The frequency with which the metadata is being rebuilt is
astronomical. E.g.
http://download.cloudstack.org/centos/8/4.15/repodata/ looks like it is
being fully rebuilt every hour, though the RPMs contained within haven't
been updated in over a month. Was this supposed to have a
--retain-old-md or --retain-old-md-by-age flag? The default for things
like RHEL 8 is 48 hours for metadata expiry, so re-generating the entire
repo every hour can (and does) cause caching issues.

  2. The metadata contained in
http://download.cloudstack.org/centos/8/4.15/repodata/repomd.xml makes
it virtually impossible to mirror since it points the  tag at
http://cloudstack.apt-get.eu/
  a. Most RPM repos (E.g. The CentOS official ones, just point the
 tag to the repodata/> of the data type without external
links via relative URI. (See
http://mirror.centos.org/centos-8/8.3.2011/BaseOS/x86_64/os/repodata/repomd.xml
, for example.)
  b. Putting the xml:base in there effectively makes it not a
mirror since anyone pointed at their local mirror will actually redirect
to whatever xml:base is set to.

  FWIW: It looks like cloudstack.apt-get.eu and
download.cloudstack.org are *probably* the same host, which likely means
the xml:base being set at all may not actually be doing anything useful.


  Please pardon my ignorance if these technical configurations are
intentional and have already been discussed as I am new poster to this
thread.


Thanks,
-Nathan McGarvey

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue






rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue