On Sun, May 13, 2018 at 11:25 AM Jeremy Stanley <fu...@yuggoth.org> wrote:
> On 2018-05-13 08:25:25 -0600 (-0600), Wesley Hayutin wrote: > [...] > > We need to in coordination with the infra team be able to pin / lock > > content for production check and gate jobs while also have the ability to > > stage new content e.g. centos 7.5 with experimental or periodic jobs. > [...] > > It looks like adjustments would be needed to DIB's centos-minimal > element if we want to be able to pin it to specific minor releases. > However, having to rotate out images in the fashion described would > be a fair amount of manual effort and seems like it would violate > our support expectations in governance if we end up pinning to older > minor versions (for major LTS versions on the other hand, we expect > to undergo this level of coordination but they come at a much slower > pace with a lot more advance warning). If we need to add controlled > roll-out of CentOS minor version updates, this is really no better > than Fedora from the Infra team's perspective and we've already said > we can't make stable branch testing guarantees for Fedora due to the > complexity involved in using different releases for each branch and > the need to support our stable branches longer than the distros are > supporting the releases on which we're testing. > This is good insight Jeremy, thanks for replying. > > For example, how long would the distro maintainers have committed to > supporting RHEL 7.4 after 7.5 was released? Longer than we're > committing to extended maintenance on our stable/queens branches? Or > would you expect projects to still continue to backport support for > these minor platform bumps to all their stable branches too? And > what sort of grace period should we give them before we take away > the old versions? Also, how many minor versions of CentOS should we > expect to end up maintaining in parallel? (Remember, every > additional image means that much extra time to build and upload to > all our providers, as well as that much more storage on our builders > and in our Glance quotas.) > -- > Jeremy Stanley > I think you may be describing a level of support that is far greater than what I was thinking. I also don't want to tax the infra team w/ n+ versions of the baseos to support. I do think it would be helpful to say have a one week change window where folks are given the opportunity to preflight check a new image and the potential impact on the job workflow the updated image may have. If I could update or create a non-voting job w/ the new image that would provide two things. 1. The first is the head's up, this new minor version of centos is coming into the system and you have $x days to deal with it. 2. The ability to build a few non-voting jobs w/ the new image to see what kind of impact it has on the workflow and deployments. In this case the updated 7.5 CentOS image worked fine w/ TripleO, however it did cause our gates to go red because.. a. when we update containers w/ zuul dependendencies all the base-os updates were pulled in and jobs timed out. b. a kernel bug workaround with virt-customize failed to work due the kernel packages changed ( 3rd party job ) c. the containers we use were not yet at CentOS 7.5 but the bm image was causing issues w/ pacemaker. d. there may be a few more that I am forgetting, but hopefully the point is made. We can fix a lot of the issues and I'm not blaming anyone because if we (tripleo ) thought of all the corner cases with our workflow we would have been able to avoid some of these issues. However it does seem like we get hit by $something every time we update a minor version of the baseos. My preference would be to have a heads up and work through the issues than to go immediately red and unable to merge patches. I don't know if other teams get impacted in similiar ways, and I understand this is a big ship and updating CentOS may work just fine for everyone else. Thanks all for your time and effort! > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
