Re: [rdo-dev] [Infra] images server cleanup
If the directory structure is the same across all directories and releases, is there a reason why we couldn't simply run a cron on the machine that woule regularly delete older images ? The migration from the previous (fedora) machine on OS1 to the new CentOS server on RDO Cloud was largely manual and there wasn't any playbooks involved. We'd like to run full automation like upstream -infrastructure does. This would allow anyone to submit a change to our playbooks, they'd be reviewed and applied automatically. Setting up this cron could be one part of the tasks involved in setting up the image server. David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Tue, Oct 24, 2017 at 10:55 AM, Gabriele Cerami wrote: > Hi, > > we'd like to participate actively in cleaning up after ourselves the > images we upload at each tripleo promotion. We are planning to do so > also for the container images in dockerhub, so part of this process has > to be done anyway. (maybe we should also do it in rdoregistry) > Since our access to the server is limited to sftp, we are thinking about > using paramiko library in our promoter script, to get the list of hashes > uploaded and their mtimes, so we can delete the oldest ones. > > Is there any better solution ? > > Thanks > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: rdo-list-unsubscr...@redhat.com > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: rdo-list-unsubscr...@redhat.com
[rdo-dev] Maintenance on the RDO container registry tonight at 12AM UTC
Hi, We'll be doing a short maintenance on the RDO container registry tonight at 12AM UTC (night from wednesday to thursday) in order to grow the volume where the container images are hosted. We'll try to coordinate around the status of the periodic jobs on review.rdoproject.org's Zuul but there's no guarantee it won't be without impact. Thanks, David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: rdo-list-unsubscr...@redhat.com
Re: [rdo-dev] Maintenance on the RDO container registry tonight at 12AM UTC
The maintenance took longer than anticipated due to unforeseen issues but is now completed successfully. Please let us know if you see any issues with the registry. David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Wed, Oct 25, 2017 at 8:45 AM, David Moreau Simard wrote: > Hi, > > We'll be doing a short maintenance on the RDO container registry tonight > at 12AM UTC (night from wednesday to thursday) in order to grow the volume > where the container images are hosted. > We'll try to coordinate around the status of the periodic jobs on > review.rdoproject.org's Zuul but there's no guarantee it won't be without > impact. > > Thanks, > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: rdo-list-unsubscr...@redhat.com
Re: [rdo-dev] [Infra] images server cleanup
Has there been any progress on this ? I'm still fairly manually cleaning up older images because the server keeps filling up. I had planned additional storage for the server but it might not have been necessary because I heard about only keeping a few images worth of backlog. Pike is always the worst offender: == # du -sh /var/www/html/images/* |sort -h 8.3G ./aarch64 9.9G ./ooo-snap 66G ./master 74G ./ocata 93G ./newton 193G ./pike == # du -sh /var/www/html/images/pike/rdo_trunk/* |sort -h 0 /var/www/html/images/pike/rdo_trunk/current-tripleo 0 /var/www/html/images/pike/rdo_trunk/current-tripleo-rdo 0 /var/www/html/images/pike/rdo_trunk/tripleo-ci-testing 1.6G /var/www/html/images/pike/rdo_trunk/tripleo-upstream 4.3G /var/www/html/images/pike/rdo_trunk/old-tripleo 8.5G /var/www/html/images/pike/rdo_trunk/0712ed3b8c6193ca4978becf70da62c6c31edabc_90cbfd4f 8.5G /var/www/html/images/pike/rdo_trunk/0de7665e14f222802fbed40fa7df93b4a4082b2d_90cbfd4f 8.5G /var/www/html/images/pike/rdo_trunk/480e79b7a3d2f0f6e6e22b92c6289426352d492c_c2957bbf 8.5G /var/www/html/images/pike/rdo_trunk/60d6e87cac10ff1f95a028c6176e768214ec8b77_9e72cb29 8.5G /var/www/html/images/pike/rdo_trunk/6beba54a71510525d5bbc4956d20d27bffa982e5_75873c3c 8.5G /var/www/html/images/pike/rdo_trunk/6d54c627703522921f41b5a83548380f1961034b_5b18c6af 8.5G /var/www/html/images/pike/rdo_trunk/75baddd6522aa86bd9028258937709c828fa1404_9e324686 8.5G /var/www/html/images/pike/rdo_trunk/8ef8f2bc46ac58385c6f92fbc9812ab6804a7ed2_4b120b84 8.5G /var/www/html/images/pike/rdo_trunk/a2369c6e219fe50fb0100806479009055ada73dc_566fe0ed 8.5G /var/www/html/images/pike/rdo_trunk/cf3665fe1c60d43aa39f1880e427875c9c571058_5b18c6af 8.5G /var/www/html/images/pike/rdo_trunk/d335965eb848cfde5cc06136b5b2fcc6b436a419_7941156c 8.5G /var/www/html/images/pike/rdo_trunk/ec8fc5d5154cab4b5167b917b11056d4bff4ef06_37239c88 8.5G /var/www/html/images/pike/rdo_trunk/f9a2508318c8e6b2c6083f1fd8f7199aba6fe1a4_0a2693a1 8.5G /var/www/html/images/pike/rdo_trunk/fd979d95d613e40be228695c0471c73cf9a5e3f4_9e324686 8.6G /var/www/html/images/pike/rdo_trunk/3c59aa392805d862096ed8ed6d9dbe4ee72f0630_e400a1b4 8.6G /var/www/html/images/pike/rdo_trunk/6bef899ed13e0dcc5ba6a99bc1859fb77682bb4c_566fe0ed 8.6G /var/www/html/images/pike/rdo_trunk/7153e0cbc5b0e6433313a3bc6051b2c0775d3804_7df0efc2 8.6G /var/www/html/images/pike/rdo_trunk/fe2afcb87218af6a3523be5a885d260ec54d24a5_31d058df 9.4G /var/www/html/images/pike/rdo_trunk/9adfb9df52a0c22da87d528da14a32d2c2b516b7_0a2693a1 == On another note, the undercloud.qcow2 image went from 2.8GB in Newton to 6.9GB in Pike... is that legitimate ? There's no bloat in there ? David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Tue, Oct 24, 2017 at 12:25 PM, David Moreau Simard wrote: > If the directory structure is the same across all directories and > releases, is there a reason why we couldn't simply run a cron on the > machine that woule regularly delete older images ? > > The migration from the previous (fedora) machine on OS1 to the new CentOS > server on RDO Cloud was largely manual and there wasn't any playbooks > involved. > We'd like to run full automation like upstream -infrastructure does. This > would allow anyone to submit a change to our playbooks, they'd be reviewed > and applied automatically. > > Setting up this cron could be one part of the tasks involved in setting up > the image server. > > > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > > On Tue, Oct 24, 2017 at 10:55 AM, Gabriele Cerami > wrote: > >> Hi, >> >> we'd like to participate actively in cleaning up after ourselves the >> images we upload at each tripleo promotion. We are planning to do so >> also for the container images in dockerhub, so part of this process has >> to be done anyway. (maybe we should also do it in rdoregistry) >> Since our access to the server is limited to sftp, we are thinking about >> using paramiko library in our promoter script, to get the list of hashes >> uploaded and their mtimes, so we can delete the oldest ones. >> >> Is there any better solution ? >> >> Thanks >> ___ >> dev mailing list >> dev@lists.rdoproject.org >> http://lists.rdoproject.org/mailman/listinfo/dev >> >> To unsubscribe: rdo-list-unsubscr...@redhat.com >> > > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: rdo-list-unsubscr...@redhat.com
[rdo-dev] Maintenance on logs.rdoproject.org
Hi, Unless there are any objections, we would like to proceed with a maintenance on the logs.rdoproject.org server during a low-traffic period on Sunday November 19th at 23:00 UTC. We are doing this at what is hopefully one of the most quiet period because jobs attempting to upload logs during this period will fail this step. In theory the maintenance should be very short, we're upgrading packages and changing root partition mount parameters. However, an unexpected event such as need to run a FSCK or a Selinux relabel could keep the server offline for a longer period of time. David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Maintenance on logs.rdoproject.org
Maintenance is complete, please let us know here or on #rdo if you notice anything wrong with the log server. Thanks. David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Fri, Nov 17, 2017 at 3:19 PM, David Moreau Simard wrote: > Hi, > > Unless there are any objections, we would like to proceed with a > maintenance on the logs.rdoproject.org server during a low-traffic period > on Sunday November 19th at 23:00 UTC. > We are doing this at what is hopefully one of the most quiet period > because jobs attempting to upload logs during this period will fail this > step. > > In theory the maintenance should be very short, we're upgrading packages > and changing root partition mount parameters. > However, an unexpected event such as need to run a FSCK or a Selinux > relabel could keep the server offline for a longer period of time. > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Project hangouts/videos
This makes me wonder... Would it be a good idea to do a "virtual" RDO meetup with real presenters and everything ? I'd love to hear about how people are contributing, using or deploying RDO. Maybe once per quarter ? Physical meetups are very local and this means a lot of folks probably never even get the chance to participate in an OpenStack meetup, let alone hear about RDO. They'd be recorded by default and available on YouTube ? It would be a great resource IMO. David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Nov 22, 2017 8:32 AM, "Rich Bowen" wrote: > Long ago, we used to do monthly (or even more frequent) hangouts covering > a variety of topics around RDO and OpenStack in general. This feel by the > wayside at some point. > > I'd like to start doing those again, following the same model as the > interviews from the PTG last month - https://www.youtube.com/watch? > v=5kT-Sv3rkTw&list=PLOuHvpVx7kYksG0NFaCaQsSkrUlj3Oq4S if you haven't seen > them yet. > > I'll be reaching out to the various project PTLs, but wanted to start with > this list here to get started. > > If you would like to talk about your work in OpenStack Upstream - 30 > minutes or less - please let me know, and I'll begin to draw up a schedule. > I'd like to start with one every two weeks, and as we'll be recording them > they can happen any time of day that's convenient for you - ie, not > necessarily live events that people will "attend". > > Thanks. > > -- > Rich Bowen: Community Architect > rbo...@redhat.com > @rbowen // @RDOCommunity // @CentOSProject > 1 859 351 9166 > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscr...@lists.rdoproject.org > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Project hangouts/videos
Meetups are usually around 2 hours from experience ? No longer than that. We can do three 30 minute time slots and allow an intro at the beginning and then some break time inbetween ? Organizing something longer than that would be tedious I think. And yeah, it'd be cheap -- no room, food or swag required ! David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Wed, Nov 22, 2017 at 8:46 AM, Rich Bowen wrote: > On 11/22/2017 08:37 AM, David Moreau Simard wrote: > >> This makes me wonder... >> >> Would it be a good idea to do a "virtual" RDO meetup with real presenters >> and everything ? >> >> I'd love to hear about how people are contributing, using or deploying >> RDO. >> >> Maybe once per quarter ? >> >> Physical meetups are very local and this means a lot of folks probably >> never even get the chance to participate in an OpenStack meetup, let alone >> hear about RDO. >> >> They'd be recorded by default and available on YouTube ? It would be a >> great resource IMO. >> >> > This sounds like a great idea, and should be fairly cheap/easy to arrange. > > Are you thinking of a full-day event, like a mini-conference, or something > shorter like an hour or two? > > --Rich > > On Nov 22, 2017 8:32 AM, "Rich Bowen" > rbo...@redhat.com>> wrote: >> >> Long ago, we used to do monthly (or even more frequent) hangouts >> covering a variety of topics around RDO and OpenStack in general. >> This feel by the wayside at some point. >> >> I'd like to start doing those again, following the same model as the >> interviews from the PTG last month - >> https://www.youtube.com/watch?v=5kT-Sv3rkTw&list=PLOuHvpVx7k >> YksG0NFaCaQsSkrUlj3Oq4S >> <https://www.youtube.com/watch?v=5kT-Sv3rkTw&list=PLOuHvpVx7 >> kYksG0NFaCaQsSkrUlj3Oq4S> >> if you haven't seen them yet. >> >> I'll be reaching out to the various project PTLs, but wanted to >> start with this list here to get started. >> >> If you would like to talk about your work in OpenStack Upstream - 30 >> minutes or less - please let me know, and I'll begin to draw up a >> schedule. I'd like to start with one every two weeks, and as we'll >> be recording them they can happen any time of day that's convenient >> for you - ie, not necessarily live events that people will "attend". >> >> Thanks. >> >> -- Rich Bowen: Community Architect >> rbo...@redhat.com <mailto:rbo...@redhat.com> >> @rbowen // @RDOCommunity // @CentOSProject >> 1 859 351 9166 >> ___ >> dev mailing list >> dev@lists.rdoproject.org <mailto:dev@lists.rdoproject.org> >> http://lists.rdoproject.org/mailman/listinfo/dev >> <http://lists.rdoproject.org/mailman/listinfo/dev> >> >> To unsubscribe: dev-unsubscr...@lists.rdoproject.org >> <mailto:dev-unsubscr...@lists.rdoproject.org> >> >> > > -- > Rich Bowen: Community Architect > rbo...@redhat.com > @rbowen // @RDOCommunity // @CentOSProject > 1 859 351 9166 > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] [Infra] images server cleanup
Since there hasn't been any progress on this, I'll implement a basic conservative cron on the image server for the time being in order to automatically delete older images. It won't be any different than the logic I've been using to clean things up manually so far. It won't have any logic around "keep 'N' last promotions", it will be based on an amount of days -- excluding the symlinked images. Please let me know once you have something else ready to use and I'll disable the cron. Thanks. David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Tue, Oct 31, 2017 at 8:42 AM, David Moreau Simard wrote: > Has there been any progress on this ? > > I'm still fairly manually cleaning up older images because the server > keeps filling up. > I had planned additional storage for the server but it might not have been > necessary because I heard about only keeping a few images worth of backlog. > > Pike is always the worst offender: > == > # du -sh /var/www/html/images/* |sort -h > 8.3G ./aarch64 > 9.9G ./ooo-snap > 66G ./master > 74G ./ocata > 93G ./newton > 193G ./pike > == > # du -sh /var/www/html/images/pike/rdo_trunk/* |sort -h > 0 /var/www/html/images/pike/rdo_trunk/current-tripleo > 0 /var/www/html/images/pike/rdo_trunk/current-tripleo-rdo > 0 /var/www/html/images/pike/rdo_trunk/tripleo-ci-testing > 1.6G /var/www/html/images/pike/rdo_trunk/tripleo-upstream > 4.3G /var/www/html/images/pike/rdo_trunk/old-tripleo > 8.5G /var/www/html/images/pike/rdo_trunk/0712ed3b8c6193ca4978becf70da62 > c6c31edabc_90cbfd4f > 8.5G /var/www/html/images/pike/rdo_trunk/0de7665e14f222802fbed40fa7df93 > b4a4082b2d_90cbfd4f > 8.5G /var/www/html/images/pike/rdo_trunk/480e79b7a3d2f0f6e6e22b92c62894 > 26352d492c_c2957bbf > 8.5G /var/www/html/images/pike/rdo_trunk/60d6e87cac10ff1f95a028c6176e76 > 8214ec8b77_9e72cb29 > 8.5G /var/www/html/images/pike/rdo_trunk/6beba54a71510525d5bbc4956d20d2 > 7bffa982e5_75873c3c > 8.5G /var/www/html/images/pike/rdo_trunk/6d54c627703522921f41b5a8354838 > 0f1961034b_5b18c6af > 8.5G /var/www/html/images/pike/rdo_trunk/75baddd6522aa86bd9028258937709 > c828fa1404_9e324686 > 8.5G /var/www/html/images/pike/rdo_trunk/8ef8f2bc46ac58385c6f92fbc9812a > b6804a7ed2_4b120b84 > 8.5G /var/www/html/images/pike/rdo_trunk/a2369c6e219fe50fb0100806479009 > 055ada73dc_566fe0ed > 8.5G /var/www/html/images/pike/rdo_trunk/cf3665fe1c60d43aa39f1880e42787 > 5c9c571058_5b18c6af > 8.5G /var/www/html/images/pike/rdo_trunk/d335965eb848cfde5cc06136b5b2fc > c6b436a419_7941156c > 8.5G /var/www/html/images/pike/rdo_trunk/ec8fc5d5154cab4b5167b917b11056 > d4bff4ef06_37239c88 > 8.5G /var/www/html/images/pike/rdo_trunk/f9a2508318c8e6b2c6083f1fd8f719 > 9aba6fe1a4_0a2693a1 > 8.5G /var/www/html/images/pike/rdo_trunk/fd979d95d613e40be228695c0471c7 > 3cf9a5e3f4_9e324686 > 8.6G /var/www/html/images/pike/rdo_trunk/3c59aa392805d862096ed8ed6d9dbe > 4ee72f0630_e400a1b4 > 8.6G /var/www/html/images/pike/rdo_trunk/6bef899ed13e0dcc5ba6a99bc1859f > b77682bb4c_566fe0ed > 8.6G /var/www/html/images/pike/rdo_trunk/7153e0cbc5b0e6433313a3bc6051b2 > c0775d3804_7df0efc2 > 8.6G /var/www/html/images/pike/rdo_trunk/fe2afcb87218af6a3523be5a885d26 > 0ec54d24a5_31d058df > 9.4G /var/www/html/images/pike/rdo_trunk/9adfb9df52a0c22da87d528da14a32 > d2c2b516b7_0a2693a1 > == > > On another note, the undercloud.qcow2 image went from 2.8GB in Newton to > 6.9GB in Pike... is that legitimate ? There's no bloat in there ? > > > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > > On Tue, Oct 24, 2017 at 12:25 PM, David Moreau Simard > wrote: > >> If the directory structure is the same across all directories and >> releases, is there a reason why we couldn't simply run a cron on the >> machine that woule regularly delete older images ? >> >> The migration from the previous (fedora) machine on OS1 to the new CentOS >> server on RDO Cloud was largely manual and there wasn't any playbooks >> involved. >> We'd like to run full automation like upstream -infrastructure does. This >> would allow anyone to submit a change to our playbooks, they'd be reviewed >> and applied automatically. >> >> Setting up this cron could be one part of the tasks involved in setting >> up the image server. >> >> >> >> David Moreau Simard >> Senior Software Engineer | OpenStack RDO >> >> dmsimard = [irc, github, twitter] >> >> On Tue, Oct 24, 2017 at 10:55 AM, Gabriele Cerami >> wrote: >> >>> Hi, >>> >>> we'd like to participate activ
Re: [rdo-dev] [all] Revisiting RDO Technical Definition of Done
On Wed, Nov 29, 2017 at 9:32 AM, Haïkel wrote: > > - CI promotion GA criteria is changed from Jenkins pipeline to the > > list of jobs running with RPM packages directly, initial set would be > > all weirdo jobs running in [3] > > I'd like to ensure that TripleO CI is still monitored closely during > the development cycle. > So it can be a non-blocking criteria for GA. > > As Javier noticed it means that our jobs will be based upon POI and > packstack. It should encourage > us to work with other installers supporting "raw" packages to make > sure that we will be able to test our > artefacts long-term. > > > - TripleO jobs would not be part of RDO GA criteria since TripelO now > > requires containers which RDO will not ship.TripleO promotion CI will > > continue running with containers built with RDO Trunk packages. > > > > Question is to know if upstream is okay with shipping containers images > using > our trunk packages. Otherwise ack. > All deployment projects are cycle-trailing [1] which means OpenStack cuts a release on day 0 and projects such as TripleO, Kolla, Packstack and Puppet-OpenStack can lag behind quite a bit , sometimes more than a week. This puts us in a very odd chicken-and-egg scenario where the deployment projects aren't ready but we are. RDO's job is to package and ship the signed tarballs as delivered by the OpenStack release. Whether or not TripleO, Packstack or Puppet-OpenStack works with the release packages is pretty unlikely to be because of problems with RDO *packages*. The content of the tarballs isn't going to change. If we end up finding a legit bug in the packaging (spec files, mirrors, etc.), these can be hotfixed easily. IMO, when comes release time, installers should be used to sanity check the *repositories* to ensure there are no missing packages, that the packages are at the right versions, etc. They've actually been quite helpful with that in the past. If there are installer bugs, I would not block the release of RDO. [1]: https://releases.openstack.org/reference/release_models.html#cycle-trailing David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] [outage] RDO Container Registry
Hi, While we monitor the disk space utilization of trunk.registry.rdoproject.org, the alerts for it were silenced due to an ongoing false positive. Last November 28th, we pruned the metadata of ~5000 image tags [1] >7 days old after which we were supposed to prune the (now orphaned) blobs. The blobs were not deleted and this lead to the registry partition running out of disk space. Container pushes from approximately the last 48 hours may have failed due as a result of the issue. We're currently pruning the orphaned blobs and pushes should work once enough free space is available. We'll address the false positive on the monitoring alert ASAP and we hope to automate the pruning process in the future to prevent this from re-ocurring. Let me know if you have any questions, [1]: https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] [outage] Recurring Nodepool problems in review.rdoproject.org
Hi, Last night at around 02:00UTC, we've had a recurrence of an issue with the nodepool instance in review.rdoproject.org where it gets stuck and quite literally stops doing anything until it is restarted. This issue leads to jobs appear queued in Zuul without any running jobs since there are no nodes being provided by Nodepool. This time corellates with a ticket opened with the RDO Cloud operations regarding heavily degraded disk performance causing flapping alerts from our monitoring. Nodepool was restarted around 09:15UTC, time at which jobs started running again. After rigorously troubleshooting what happened, we found and fixed several issues which compounded each other: #1 The clouds.yaml file for Nodepool did not specify an api-timeout parameter This parameter defaults to None [1][2] and this caused Shade to potentially hang for very long periods of time. We've added the api-timeout parameter to our clouds.yaml file for each Nodepool provider and this should help Nodepool (and shade) recover in a situation where the API might be temporarily unresponsive. We sent a patch to address this in Software Factory's implementation of Nodepool. [3] #2 The clouds.yaml file for Nodepool did not specify caching parameters Caching parameters allows Shade and Nodepool to alleviate the load on the API servers by reducing the amount of calls and we were not caching anything. We aligned our configuration for caching with the same settings currently used upstream and to provide support for it [4]. #3 Shade queries the API to get a list of ports, then queries the API to determine if each port is a floating IP In what is currently considered a bug described here [5], Shade will end up doing 7 API calls for a server with 6 ports (ex: baremetal OVB node): - 1x to get the list of ports - 6x to check if any of the ports are floating IPs This makes some functions Nodepool routinely uses, such as list_servers(), take a considerable amount of time and put a lot of strain on the API server. While troubleshooting this, we found that even though caching was enabled, Shade wasn't using it and this was addressed in a patch written today [6]. We haven't quite figured out a fix to prevent this problematic behavior from happening when caching is disabled but we have several ideas. For the time being, we have worked around the issue by manually applying this patch to our shade installation and enabling caching in clouds.yaml. We have good reasons to believe that these three issues were either responsible or largely contributed to the recurring problem. If it does happen again, we have a procedure to get Nodepool to dump what exactly is going on. [1]: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/defaults.py?id=0d062b7d6fb29df035508117c15770c6846f7df4#n41 [2]: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/openstackcloud.py?id=faaf5f024bd57162466c273dde370068ce55decf#n164 [3]: https://softwarefactory-project.io/r/10542 [4]: https://softwarefactory-project.io/r/10541 [5]: https://storyboard.openstack.org/#!/story/2001394 [6]: https://review.openstack.org/#/c/526127/ David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] [outage] New package builds are tempoarily paused on trunk.rdoproject.org
Hi, There is an ongoing maintenance on the primary cloud provider for RDO's infrastructure and while the mirrors on trunk.rdoproject.org are *not* affected by this maintenance, the node responsible for new packages is. The intermittent network instability can lead package builds or the mirror synchronization to fail which can lead to other problems such as erroneously missing packages [1]. For the time being, we've resolved the problems caused by this instability and have paused new builds so that we do not generate any further inconsistency. If jobs have failed due to HTTP 403 (Forbidden) or HTTP 404 (Not found) errors on packages provided by the RDO trunk repositories, please try again. We will stay up to date on maintenance and re-enable package builds once we are confident in the status of the network. Thanks. [1]: https://bugs.launchpad.net/tripleo/+bug/1737611 David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Maintenance on review.rdoproject.org *tonight* December 20th 23:00 UTC
Hi, Our apologies for the short notice but review.rdoproject.org will be undergoing maintenance tonight at 23:00 UTC in order to update to operating system and software factory packages to the latest versions. We are planning one hour for the maintenance but it could take longer if we run into unexpected complications. There may brief moments of downtime as we will need to reboot the different instances of the cluster at least once. Please note that while this update will enable us to activate Zuul v3, it will *not* be implemented tonight. Zuul v3 will be installed at a later date, at the very least after the upcoming holiday period. If you have any questions or notice any unexpected issues, please reach out to us on #rdo on IRC or by replying to this thread. Thanks ! David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Maintenance on review.rdoproject.org *tonight* December 20th 23:00 UTC
The maintenance is completed and the upgrade was successful. The update was completed relatively quickly but it took some time to troubleshoot the Nodepool and Jenkins configuration to work properly with the new version of Software Factory. For this reason, it is possible that you might need to recheck some patches if you have gotten errors such as: - zuul-cloner not found - NOT_REGISTERED Everything is currently functional so far as we can tell. If you see any problems, please notify us on #rdo or through this thread. Thanks ! David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Wed, Dec 20, 2017 at 10:58 AM, David Moreau Simard wrote: > Hi, > > Our apologies for the short notice but review.rdoproject.org will be > undergoing maintenance tonight at 23:00 UTC in order to update to operating > system and software factory packages to the latest versions. > We are planning one hour for the maintenance but it could take longer if > we run into unexpected complications. > > There may brief moments of downtime as we will need to reboot the > different instances of the cluster at least once. > > Please note that while this update will enable us to activate Zuul v3, it > will *not* be implemented tonight. > Zuul v3 will be installed at a later date, at the very least after the > upcoming holiday period. > > If you have any questions or notice any unexpected issues, please reach > out to us on #rdo on IRC or by replying to this thread. > > Thanks ! > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Rolling updates and reboots: Patching kernels with recent CPU CVEs
Hi, The updated CentOS kernel packages with fixes for the recent CPU CVEs have been made available and as such we will proceed with updating all the servers in RDO's infrastructure. There might be a brief moment on unavailability as certain servers reboot. Please reach out to us in #rdo if you notice any problems. Thanks, David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Rolling updates and reboots: Patching kernels with recent CPU CVEs
Updates and reboots have been completed for the bulk of the servers part of the RDO infrastructure, including (but not limited to): - www.rdoproject.org - lists.rdoproject.org - review.rdoproject.org servers - trunk build, database and repository servers - registry.rdoproject.org - logs.rdoproject.org - images.rdoproject.org - backup server There has been little to no impact that I could tell throughout the process of updating the servers: - Some false alarm FTBFS from trunk.rdoproject.org during the updates - Some TripleO jobs have had to be interrupted on review.rdoproject.org (they were stuck >5hrs, for example) - Some TripleO jobs from review.rdoproject.org may have had a heat stack create failed after review.rdo was rebooted due to the amount of jobs triggering simultaneously Please let us know here or on #rdo if you notice anything unusual. Thanks ! David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Thu, Jan 4, 2018 at 12:13 PM, David Moreau Simard wrote: > Hi, > > The updated CentOS kernel packages with fixes for the recent CPU CVEs have > been made available and as such we will proceed with updating all the > servers in RDO's infrastructure. > There might be a brief moment on unavailability as certain servers reboot. > > Please reach out to us in #rdo if you notice any problems. > > Thanks, > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Rolling updates and reboots: Patching kernels with recent CPU CVEs
Also take note that the cloud used to host the RDO infrastructure will also be undergoing maintenance updates shortly and while the operators will try to limit the impact of the maintenance, there might be some periods of unavailability. David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Thu, Jan 4, 2018 at 6:47 PM, David Moreau Simard wrote: > Updates and reboots have been completed for the bulk of the servers part > of the RDO infrastructure, including (but not limited to): > - www.rdoproject.org > - lists.rdoproject.org > - review.rdoproject.org servers > - trunk build, database and repository servers > - registry.rdoproject.org > - logs.rdoproject.org > - images.rdoproject.org > - backup server > > There has been little to no impact that I could tell throughout the > process of updating the servers: > - Some false alarm FTBFS from trunk.rdoproject.org during the updates > - Some TripleO jobs have had to be interrupted on review.rdoproject.org > (they were stuck >5hrs, for example) > - Some TripleO jobs from review.rdoproject.org may have had a heat stack > create failed after review.rdo was rebooted due to the amount of jobs > triggering simultaneously > > Please let us know here or on #rdo if you notice anything unusual. > > Thanks ! > > > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] > > On Thu, Jan 4, 2018 at 12:13 PM, David Moreau Simard > wrote: > >> Hi, >> >> The updated CentOS kernel packages with fixes for the recent CPU CVEs >> have been made available and as such we will proceed with updating all the >> servers in RDO's infrastructure. >> There might be a brief moment on unavailability as certain servers reboot. >> >> Please reach out to us in #rdo if you notice any problems. >> >> Thanks, >> >> David Moreau Simard >> Senior Software Engineer | OpenStack RDO >> >> dmsimard = [irc, github, twitter] >> > > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Graphite and Grafana in RDO-Cloud
There are already plans [1] to add the software factory implementation of Grafana on review.rdoproject.org, you can see what it looks like on softwarefactory-project.io [2]. The backend to this grafana implementation is currently influxdb, not graphite. However, there are ongoing discussions to either both graphite and influxdb simultaneously or optionally either. We're interested in leveraging this influxdb (or graphite) and grafana implementation for monitoring data in general (uptime, resources, disk space, load, etc.) so our goals align here. We both agree that using graphite would be a plus in order to re-use the same queries in the grafana dashboard but at the same time, influxdb is more "modern" and easier to work with -- this is why we might end up deploying both, we'll see. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1514086 [2]: https://softwarefactory-project.io/grafana/ David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Fri, Jan 5, 2018 at 12:13 PM, Wesley Hayutin wrote: > Greetings, > > At the end of 2017, a number of the upstream multinode scenario jobs > started to run over our required deployment times [1]. In an effort to > better understand the performance of the deployment and CI the tripleo > cores requested that a Graphite and Grafana server be stood up such that we > can analyze the core issues more effectively. > > There is a certain amount of urgency with the issue as our upstream > coverage is impacted. The TripleO-CI team is working on the deployment of > both tools in a dev-ops style in RDO-Cloud this sprint. Nothing yet has > been deployed. > > The TripleO CI team is also working with upstream infra to send metric and > data to the upstream Graphite and Grafana servers. It is not clear yet if > we have permission or access to the upstream tools. > > I wanted to publically announce this work to the RDO infra community to > inform and to gather any feedback anyone may have. There are two scopes of > work here, the initial tooling to stand up the infra and the longer term > maintenance of the tools. Perhaps there are plans to build these into RDO > SF already.. etc. > > Please reply with your comments and concerns. > Thank you! > > > [1] https://github.com/openstack-infra/tripleo-ci/commit/ > 7a2edf70eccfc7002d26fd1ce1eef803ce8d0ba8 > > > > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscr...@lists.rdoproject.org > > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] RDO's infrastructure server metrics are now available
Hi, We have historically been monitoring RDO's infrastructure through Sensu and it has served us well to pre-emptively detect issues and maximize our uptime. At some point, Software Factory [1] grew an implementation of Grafana, InfluxDB and Telegraf in order to monitor the health of the servers, not unlike how upstream's openstack-infra leverages cacti. This implementation was meant to eventually host graphs such as the ones for Zuul and Nodepool upstream. While there are still details to be ironed out for the Zuul and Nodepool data collection, there was nothing preventing us from just deploying telegraf everywhere just for the general server metrics. It's one standalone package and one configuration file, that's it. Originally, we had been thinking about feeding the Sensu metric data to Influxdb [2]... but why even bother if it's there for free in Software Factory ? So here we are. The metrics are now available here: https://review.rdoproject.org/grafana We will use this as a foundation to improve visibility into RDO's infrastructure, make it more "open" and accessible in the future. We're not getting rid of Sensu although we may narrow it's scope to keep some of the more complex service and miscellaneous monitoring that we need to be doing. We'll see what time has in store for us. Let me know if you have any questions ! [1]: https://softwarefactory-project.io [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1514089 David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Maintenance: review.rdoproject.org tonight @ 23:30UTC
Hi, We need to restart Jenkins on review.rdoproject.org to make new configuration parameters effective. We hope this new configuration will alleviate some of the performance and slave disconnection issues we have been seeing. In order to be able to restart Jenkins without impacting running jobs, all new jobs will be queued and we will restart Jenkins once the current jobs have finished. We hope to be able to do this around 23:30UTC tonight if jobs have all finished running, otherwise we will wait until they're complete before proceeding. Thanks, David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Maintenance: review.rdoproject.org tonight @ 23:30UTC
The maintenance is finished. Some jobs would have kept going for a few hours and were aborted. Feel free to issue a recheck if need be. Thanks, David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Thu, Jan 18, 2018 at 5:31 PM, David Moreau Simard wrote: > Hi, > > We need to restart Jenkins on review.rdoproject.org to make new > configuration parameters effective. > We hope this new configuration will alleviate some of the performance > and slave disconnection issues we have been seeing. > > In order to be able to restart Jenkins without impacting running jobs, > all new jobs will be queued and we will restart Jenkins once the > current jobs have finished. > > We hope to be able to do this around 23:30UTC tonight if jobs have all > finished running, otherwise we will wait until they're complete before > proceeding. > > Thanks, > > David Moreau Simard > Senior Software Engineer | OpenStack RDO > > dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] proposing rlandy as core
+2, Ronelle has learned a lot and is entirely deserving to be part of the core configuration group. Congrats :) David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Jan 17, 2018 11:41 AM, "Wesley Hayutin" wrote: > Greetings, > > As discussed in the RDO meeting today, I am proposing rlandy as core for > [1]. I'm not 100% sure how the access is divided up so I'm specifying the > repo itself. Ronelle has proven herself to be a quality reviewer and her > submissions are also up to the standards of core. > > Thank you all for your time in considering Ronelle as core. > > > [1] https://github.com/rdo-infra/review.rdoproject.org-config > > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscr...@lists.rdoproject.org > > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Delaying this week's RDO test days ?
Hi, We're currently a bit behind in terms of trunk repository promotions for the m3 queens milestone. We are not very confident that we can get all the issues sorted out in time for the test day so I would like to suggest we push the test days next week. Instead of February 8th/9th, we would instead run the test days on the 15th/16th. Ideally we would reach a consensus before wednesday's RDO meeting at which point it would be too late. Thanks, David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Happening now: secret, key and password rotation
Hi, Jenkins has published a list of serious vulnerabilities [1]. Please note that while we have no reasons to believe we have been compromised, we are still going to be rotating the various secrets, keys and passwords stored in Jenkins credentials as a safety precaution. For jobs using JJB, this should hopefully be transparent since the UUID of the secret will not change (but the contents of the secret will). If anything around RDO CI automation (including third party CI) breaks as a result of this, please let me know here or on IRC (dmsimard) Thanks, [1]: https://jenkins.io/security/advisory/2018-02-14/ David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] proposal to alert #rdo w/ issues from launchpad
+1, we have been trying to keep #rdo bot-spam free as much as possible. I'm not personally a fan of the bot spam in #tripleo and #rhos-dev (the one there in #rhos-dev is actually quite obnoxious, actually, but that's another topic). If we are going to want something like this, I would tend to formalize the #rdo-infra channel. I think it would make sense... For example, the config repo notifications would be sent there as well as monitoring alerts. We'd get rid of #rdo-dev because it was never meant to be a thing. The development of RDO happens on #rdo. I think we created #rdo-dev back when we implemented the monitoring and it's main purpose was just to not spam #rdo. I could see the implementation of Wes' suggestion being a proper monitoring check just like the others that we have [1]. This would allow us to control the way the notifications behave, amongst other things. I'm happy to point folks in the right direction to make this happen, everything is under code review and integration tested. [1]: https://github.com/rdo-infra/rdo-infra-playbooks/tree/master/roles/rdo-infra/sensu-client/files David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Mar 3, 2018 9:45 AM, "Alan Pevec" wrote: Hi Wes, I'd prefer to integrate those alerts into existing RDO monitoring instead of adding one more bot. We have #rdo-dev channel where infra alerts would fit better, can you show few example LPs where you those tags would be applied? Alan ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] RDO infra/TripleO-CI teams integration level on infrastructure
ers, so we can already use the log server to expose our > promotion logs ? That said, I think our existing roles are generic enough -- especially the base and monitoring roles. If you have improvements for these, feel free to submit them ! For this particular case, the logserver role that exists today [4] is actually not the one that was used to deploy our current logserver implementation which is more in line with a WIP patch here [5]. I think we need to work on describing what we need better, agree on a solution and then go from there. In this case, our need is to expose the promotion logs -- this can be provided in a number of ways ranging from our existing logstash setup to simply doing a "passthrough" of the logs over a local httpd server. In the upstream infrastructure, these kind of proposals are discussed as specs [6]. It allows everyone a chance to give their opinion which not only results in a better spec in the end but it also allows to avoid starting with bad assumptions. For example, you might draft a spec to deploy a new component in the infrastructure but it turns out we already have something exactly like that but you didn't know about it. At the end of the day, let's not forget that RDO is an open source community that anyone can participate in just like the upstream OpenStack one. You can show up tomorrow in #openstack-infra and start sending and reviewing patches if you'd like. The upstream infra-root and config-core teams are more or less volounteers that have accepted to take on this role regardless of their organization or affiliation. They do not really have any privileged or closed channels. Everything is in the open, everyone can contribute. What I'm trying to say is that if someone from outside of Red Hat comes and volunteers to help with the management of the RDO infrastructure, I'm definitely not going to turn that person down. So the scope here is not so much "integration between two teams" but instead looking at the opportunities to have everyone collaborating to improve the infrastructure in general. Hope that makes sense. [1]: https://review.rdoproject.org/r/#/c/12777/1/ci-scripts/infra-setup/roles/base_setup/base-setup.yaml [2]: https://github.com/rdo-infra/rdo-infra-playbooks [3]: https://review.rdoproject.org/r/#/q/project:rdo-infra/rdo-infra-playbooks [4]: https://github.com/rdo-infra/ansible-role-logserver [5]: https://review.rdoproject.org/r/#/c/10814/ [6]: https://github.com/openstack-infra/infra-specs David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Missing updated Nova package for Pike
Hi, Nova released 16.1 for Pike in February [1][2], there's no spec updates in the distgit repo [3] and rdoinfo is still pointing to the previous release [4]. Is there something wrong in the automation ? [1]: https://github.com/openstack/nova/releases/tag/16.1.0 [2]: https://github.com/openstack/releases/commit/46daf6fce8ef67ec9e439561dbc5376a25933f73#diff-c86c2132bd6bdc4a2d459b6c360896e1 [3]: https://review.rdoproject.org/r/#/q/project:openstack/nova-distgit [4]: https://github.com/redhat-openstack/rdoinfo/blob/db50e8cb50d7a03a6170c58bd4c7f67b6ae990f2/rdo.yml#L1964-L1965 David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] README in distgit projects
Hi, Chris Dent rightfully pointed out that we don't typically have a proper and up-to-date README in our distgit repos [1]. Can we/should we fix that ? The README could be an appropriate place to document small things like: - Link to review.rdoproject.org scoped for this repository's reviews [2] - Quickstart: git clone, edit, git review (and mention no pull requests) - Link to packager documentation - Other things Probably even something worthy of easyfix [3] ? [1]: Example: https://github.com/rdo-packages/nova-distgit [2]: Example: https://review.rdoproject.org/r/#/q/project:openstack/nova-distgit [3]: https://github.com/redhat-openstack/easyfix David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Nomination of new RDO infrastructure cores
Hi ! The actual nomination actually brings us back to January [1] and I dropped the ball about broadly communicating this. The following names might not be familiar to everyone in the RDO community but they have been instrumental in getting our production chain and CI to where it is today: - Tristan Cacqueray (tristanC) - Nicolas Hicher (nhicher) - Fabien Boucher (fbo) - Matthieu Huin (mhu) You might have heard about Software Factory [2], especially when discussing review.rdoproject.org, the RDO CI or the RDO production chain. They deserve a lot of the credit for providing us with an awesome platform for contributing to RDO. They are already familiar with the different tools and servers used by the RDO community. Granting them access to our infrastructure will help us spread the operational workload but will more importantly give them a first-person impression of the scale we are working at. I am confident that this experience would be a win/win for RDO and for Software Factory. If there are no objections, I'd like to formalize this and add their keys to our servers [3]. [1]: http://eavesdrop.openstack.org/meetings/rdo_meeting___2018_01_17/2018/rdo_meeting___2018_01_17.2018-01-17-15.00.log.html#l-59 [2]: https://softwarefactory-project.io [3]: https://review.rdoproject.org/r/13383 David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] [ppc64le] Spinning up Triple O container jobs
Michael, The cloud from which Duffy provides virtual machines is not publicly available, this means we cannot leverage it with Nodepool or Zuul. We have some tooling to request [1][2] and destroy [3][4] Duffy nodes. Do you really need a custom image or would requesting a "vanilla" ppc64le node from Duffy be enough ? [1]: https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/cico-node-get-to-ansible.sh [2]: https://github.com/rdo-infra/ci-config/blob/516411b57b378c1bf7374bcf32774e2f5dd1b2fb/jenkins/jobs/weirdo-builders.yml#L1-L6 [3]: https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/cico-node-done-from-ansible.sh [4]: https://github.com/rdo-infra/ci-config/blob/516411b57b378c1bf7374bcf32774e2f5dd1b2fb/jenkins/jobs/weirdo-defaults.yml#L68-L78 David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Fri, May 11, 2018 at 12:49 PM, Michael Turek wrote: > Hey all, > > Last week I threw up a patch to add tripleo container build jobs for ppc64le > [0]. I would really appreciate getting some eyes on it. I'm not sure how > complete it is, but I do know that we'll need: > > a) A provider for ppc64le in nodepool [1] > b) A ppc64le image in nodepool. [2] > > There's been some discussion around (a) which led to me being linked to > https://wiki.centos.org/QaWiki/CI/Multiarch but I don't think Duffy can work > with nodepool. If I'm wrong about this, please let me know. > > As for (b), we would need a node running DIB on a ppc64le node/guest > providing a ppc64le image . > > Thanks, > Mike Turek > > [0] https://review.rdoproject.org/r/#/c/13606/ > [1] https://review.rdoproject.org/r/#/c/13606/4/nodepool/nodepool.yaml@134 > [2] https://review.rdoproject.org/r/#/c/13606/3/nodepool/nodepool.yaml@145 > > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscr...@lists.rdoproject.org ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] [ppc64le] Spinning up Triple O container jobs
What are the requirements for those jobs other than ppc64le ? Do you need OVB ? Do you need heat ? If you don't need either and duffy won't work, we can consider asking the CentOS infrastructure team if we can upload custom images to the cico cloud. They have some amount of ppc64le capacity available [1]. [1]: https://wiki.centos.org/QaWiki/CI/Multiarch#head-02310f99181bff98c506d96637895e982c799226 David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] On Fri, May 11, 2018 at 1:36 PM, Michael Turek wrote: > Hey David, > > Replies inline > > > On 5/11/18 12:57 PM, David Moreau Simard wrote: >> >> Michael, >> >> The cloud from which Duffy provides virtual machines is not publicly >> available, this means we cannot leverage it with Nodepool or Zuul. > > Fair enough! I guess the alternative is to get Power nodes into the RDO > cloud? Any other ideas? >> >> >> We have some tooling to request [1][2] and destroy [3][4] Duffy nodes. >> Do you really need a custom image or would requesting a "vanilla" >> ppc64le node from Duffy be enough ? > > Not sure about this. The image used for container builds on x86_64 > (upstream-centos-7) seems to use quite a few elements [0]. I'm not clear on > why each of them are required, but I would assume a power node would need an > image built with these elements as well. > >> >> [1]: >> https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/cico-node-get-to-ansible.sh >> [2]: >> https://github.com/rdo-infra/ci-config/blob/516411b57b378c1bf7374bcf32774e2f5dd1b2fb/jenkins/jobs/weirdo-builders.yml#L1-L6 >> [3]: >> https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/cico-node-done-from-ansible.sh >> [4]: >> https://github.com/rdo-infra/ci-config/blob/516411b57b378c1bf7374bcf32774e2f5dd1b2fb/jenkins/jobs/weirdo-defaults.yml#L68-L78 >> >> David Moreau Simard >> Senior Software Engineer | OpenStack RDO >> >> dmsimard = [irc, github, twitter] >> >> >> On Fri, May 11, 2018 at 12:49 PM, Michael Turek >> wrote: >>> >>> Hey all, >>> >>> Last week I threw up a patch to add tripleo container build jobs for >>> ppc64le >>> [0]. I would really appreciate getting some eyes on it. I'm not sure how >>> complete it is, but I do know that we'll need: >>> >>> a) A provider for ppc64le in nodepool [1] >>> b) A ppc64le image in nodepool. [2] >>> >>> There's been some discussion around (a) which led to me being linked to >>> https://wiki.centos.org/QaWiki/CI/Multiarch but I don't think Duffy can >>> work >>> with nodepool. If I'm wrong about this, please let me know. >>> >>> As for (b), we would need a node running DIB on a ppc64le node/guest >>> providing a ppc64le image . >>> >>> Thanks, >>> Mike Turek >>> >>> [0] https://review.rdoproject.org/r/#/c/13606/ >>> [1] >>> https://review.rdoproject.org/r/#/c/13606/4/nodepool/nodepool.yaml@134 >>> [2] >>> https://review.rdoproject.org/r/#/c/13606/3/nodepool/nodepool.yaml@145 >>> >>> ___ >>> dev mailing list >>> dev@lists.rdoproject.org >>> http://lists.rdoproject.org/mailman/listinfo/dev >>> >>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org > > Thanks, > Mike Turek > > [0] > https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/nodepool/nodepool.yaml#L15-L27 > ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] EPEL packages needed by OpenStack-Ansible
Hi, openstack-ansible is working on adding full support for RDO [1]. They currently need EPEL for a few dependencies which aren't currently in the RDO mirrors. Unsurprisingly, things end up conflicting at one point or another because of EPEL being enabled. At first glance, the following packages are the only ones needed: - apt-cacher-ng-3-1.el7: https://koji.fedoraproject.org/koji/buildinfo?buildID=915750 - aria2-1.18.10-2.el7.1: https://koji.fedoraproject.org/koji/buildinfo?buildID=642076 - lsyncd-2.2.2-1.el7: https://koji.fedoraproject.org/koji/buildinfo?buildID=879986 - uwsgi-2.0.16-1.el7: https://koji.fedoraproject.org/koji/buildinfo?buildID=1044001 - moreutils-0.57-1.el7: https://koji.fedoraproject.org/koji/buildinfo?buildID=703215 - nginx-1.12.2-2.el7: https://koji.fedoraproject.org/koji/buildinfo?buildID=1054143 - scsi-target-utils-1.0.55-4.el7: https://koji.fedoraproject.org/koji/buildinfo?buildID=710762 Can we do straight rebuilds from EPEL for those or is it more complicated than that ? [1]: https://review.openstack.org/#/q/topic:bp/openstack-distribution-packages David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
[rdo-dev] Unscheduled updates and upgrade for registry.rdoproject.org
Hi, In light of the recent unreliability of trunk.registry.rdoproject.org, the instance was updated and resized with more RAM to account for the increased amount of images. There may have been outages earlier today between 15:00 UTC and 16:30 UTC while we were applying updates, rebooting and resizing. Please let us know if there is any issues moving forward. Thanks, David Moreau Simard dmsimard = [irc, github, twitter] ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] Proposal for administrators group update in review.rdoproject.org
+1 David Moreau Simard dmsimard = [irc, github, twitter] On Wed, Sep 11, 2019 at 10:40 AM Javier Pena wrote: > > Hi, > > I would like to propose the following changes to the Administrators group in > review.rdoproject.org [1]: > > - Add Yatin Karel (ykarel) to the group. Actually, I'm surprised he was not > already an administrator. > - Remove Frederic Lepied and Haïkel Guemar from the group. Both have moved on > to other tasks, and do not work on a daily basis in the management side. > > Please reply in the list with your votes (+1/-1). I am planning to do the > change next Friday. > > Regards, > Javier > > [1] - https://review.rdoproject.org/r/#/admin/groups/1,members > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscr...@lists.rdoproject.org ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
