Hi Daniel, Many thanks for your answer, and for the detailed list of ways to mirror the repos. The wget depth limit was just a way of testing without waiting for a whole tree sync whether this was just lftp on my side wich was failing, or really something that changed in the repos.
I tried rsync last week and it was awfully slow, and I ended up both upgrading lftp to a freshly compiled one + removing an “ignore-time” option, which fixed the issue for now – but I’m looking for a decent lftp replacement now ;). I’ll see if I can implement someting like pulp , reposync or sync-repo, but I’d prefer finding something that’s not just limited to syncing yum Regards De : puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] De la part de Daniel Dreier Envoyé : vendredi 30 juin 2017 20:57 À : puppet-users@googlegroups.com Objet : Re: [Puppet Users] apt, yum, downloads, and rsync infrastructure improvements On Fri, Jun 30, 2017 at 2:14 AM, SCHAER Frederic <frederic.sch...@cea.fr<mailto:frederic.sch...@cea.fr>> wrote: Hi, I was (up to now) mirroring the puppetlabs repositories to both : - Make sure I have a local copy in case your repos are down, or our internet link is too weak - Not hammer on your infrastructure with our servers Much appreciated! Unfortunately, we just noticed our mirroring suddenly got broken just after your changes. We use lftp, but I just tried with a recursive wget and see the same issue, depending on the max depth I ask. What I see after a simple wget –r –l1 yum.puppetlabs.com<http://yum.puppetlabs.com> is this : # ll yum.puppetlabs.com/el/6Server/<http://yum.puppetlabs.com/el/6Server/> total 28 -rw-r--r-- 1 root root 1635 Jun 28 20:01 dependencies -rw-r--r-- 1 root root 1603 Jun 28 20:01 devel -rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_lastModified.html -rw-r--r-- 1 root root 1824 Jun 28 20:01 index_by_name_reverse.html -rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_size.html -rw-r--r-- 1 root root 1592 Jun 28 20:01 PC1 -rw-r--r-- 1 root root 1618 Jun 28 20:01 products Symlinks or directories now are files, and those are html files with relative hrefs to the corresponding files/directories Symlinks don't exist anymore because we're using S3 as the backend to CloudFront, and S3 has no concept of symlinks. We're still shipping symlinks to the rsync servers, so if you "rsync -a rsync://rsync.puppet.com/packages/yum<http://rsync.puppet.com/packages/yum> /tmp/yum" you'll get the symlinks, which is a lot more space efficient than mirroring the fully-resolved filesystems via HTTPS. Is there a « now recommended » way of mirroring the repos ? (I’ve tried the lftp –dereference option to no avail…) Each of the following should work: Rsync: rsync -a rsync://rsync.puppet.com/packages/yum<http://rsync.puppet.com/packages/yum> /tmp/yum Directly from S3: aws s3 sync --exclude '*.html' s3://yum.puppetlabs.com<http://yum.puppetlabs.com> /tmp/yum HTTPS via CloudFront: wget -r https://yum.puppetlabs.com Of them, rsync and aws s3 sync are both incremental, so they'll be more bandwidth efficient and faster compared to using wget. If any of those options aren't working, I'd like to know about it. I'm seeing the same behavior you described when wget max depth is set, but I don't understand the use case where you'd limit depth on it; it seems like you'd end up with a broken repository if you didn't sync down pretty much everything. I see rsync is still sort of supported, but I have the impression you would prefer that we use your new CDN (hence, get the files through cloudfront using http …) rsync is absolutely still supported. After we discontinued it last time and got a lot of pushback, I've been working on an HA rsync. Thanks && regards Frederic Schaer De : puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com> [mailto:puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>] De la part de Daniel Dreier Envoyé : mercredi 28 juin 2017 20:26 À : puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com> Objet : [Puppet Users] apt, yum, downloads, and rsync infrastructure improvements Today we're making improvements to apt.puppetlabs.com<http://apt.puppetlabs.com>, yum.puppetlabs.com<http://yum.puppetlabs.com>, and downloads.puppetlabs.com<http://downloads.puppetlabs.com>. I don't anticipate any user-visible changes, and this notification is just to let folks know so that you can report problems to me. Specifically, we're switching out the CDN backend - it's been running off a server in Linode, and we're switching to an S3 backend, which will be faster and more reliable. If you see any problems with the apt/yum download infrastructure, please let me know. -- Daniel Dreier Technical Operations Engineer GPG: BA4379FD -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com<mailto:puppet-users+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAGk8subLmT0QDSh79Jd9KOCxfzRXOTqK4i0Padvhk3Abq0iQTg%40mail.gmail.com<https://groups.google.com/d/msgid/puppet-users/CAGk8subLmT0QDSh79Jd9KOCxfzRXOTqK4i0Padvhk3Abq0iQTg%40mail.gmail.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com<mailto:puppet-users+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA2317B21A78%40E-EXDAGE-A0.extra.cea.fr<https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA2317B21A78%40E-EXDAGE-A0.extra.cea.fr?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- Daniel Dreier Technical Operations Engineer GPG: BA4379FD -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com<mailto:puppet-users+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAGk8suZaso1PSeQUvKQ3QYKRuPhGUagBCgiOEQJmKR-eGRBFFQ%40mail.gmail.com<https://groups.google.com/d/msgid/puppet-users/CAGk8suZaso1PSeQUvKQ3QYKRuPhGUagBCgiOEQJmKR-eGRBFFQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA2317B2313E%40E-EXDAGE-A0.extra.cea.fr. For more options, visit https://groups.google.com/d/optout.
